How to capture EventSource messages while process is running?

I hail from here, where David Friend was an absolute help! However, we have encountered a problem that needs more eyes, so here's a new thread.

I have a codeigniter app that takes in user input, uses JS to fetch/POST the input to a controller, which has it processed by a model. The model needs to send progress updates to the user during the process, but it seems like the fetch promise is capturing all the messages before eventsource can get them and won't display them until the process is finished.

Is there a way to stop fetch from hoarding all the eventsource messages? Or maybe catch each one and display them as they come in and not until the process is done?

My index.php

Code:

    <form id="form1">
      <div class="form-group">
        <label>Date From -></label>
        <input type="text" class="form-control" name="date" id="magento-date" placeholder="2019-10-15 06:46:30">
      </div>
      <div class="form-group">
        <label>Date To <-</label>
        <input type="text" class="form-control" name="dateTo" id="magento-date-to" placeholder="2019-10-16 06:46:30">
      </div>

      <button id="submit" type="submit" class="btn btn-default">Submit</button>
    </form>
    <div class="progress progress-striped active">
      <div class="progress-bar" id="progress" style="width:0%"></div>
    </div>
    <div id="message"></div>

    <script>

    document.querySelector("#submit").addEventListener("click", function(event) {
      event.preventDefault();
      const url = <?= "'".base_url('classic/classicFirearmByDate', 'http')."'"; ?>;

      var myHeaders = new Headers();

      const form = new FormData(document.querySelector('#form1'));

      // The parameters we are gonna pass to the fetch function
      let fetchData = {
          method: 'POST',
          headers: myHeaders,
          body: new URLSearchParams(form)
      }

      //{method: "POST",header:myHeaders,body:{date:'2020baby'}}
      fetch(url, fetchData, message())
      .then((res) => res.text())
      .then(text => console.log(text))
      .catch((error) => console.log(error));

    }, false);
    function message() {
      var evtSource = new EventSource(<?= "'".base_url('classic/classicFirearmByDate', 'http')."'"; ?> );
      var eventList = document.querySelector('#message');

      evtSource.onopen = function () {
          console.log("Connection to server opened.");
      };

      evtSource.onmessage = function (e) {
          console.log(e);
          var newElement = document.getElementById("message");

          newElement.textContent = "message: " + e.data;
          // eventList.appendChild(newElement);
      };

      evtSource.onerror = function () {
          console.log("EventSource failed.");
      };
    }
    </script>

My controller.php

Code:

<?php
class Classic extends CI_Controller{
    . . .
    public function classicFirearmByDate() {
        $data['title'] = 'Classic Firearms';
        $sandbox = $this->input->post('sandbox');
        . . .
        $date = trim(htmlspecialchars($this->input->post('date')));
        $dateTo = trim(htmlspecialchars($this->input->post('dateTo')));
        $data['body'] = $this->classic_firearms->get_firearm_by_date($date, $dateTo, $sandbox, $this->input->post());

My model.php

Code:

<?php
class Classic_firearms extends CI_Model{
. . .
    public function get_firearm_by_date($date, $dateTo, $sandbox, $test) {
            $this->sessionId = $this->login($sandbox);
            $filter = array('complex_filter' => array(
                    array(
                            'key' => 'updated_at',
                            'value' => array(
                                    'key' => 'from',
                                    'value' => $date//'2019-10-03 09:44:30'//
                            ),
                    ),
                    array(
                            'key' => 'updated_at',
                            'value' => array(
                                    'key' => 'to',
                                    'value' => $dateTo//'2019-10-03 09:46:30'//
                            ),
                    ),
            ));

            $productId = $this->proxy->catalogProductList($this->sessionId, $filter);


            header("Content-Type: text/event-stream");
            header("Cache-Control: no-cache");
            header("Connection: keep-alive");

            if ($productId != null || $productId != [] ) {
                    . . . //Many cURL requests and data conversion
                            echo 'data:  Working on {$product_name} Product!', "\n\n";

                            // flush the output buffer and send echoed messages to the browser
                            flush();

One idea I have is to post the user inputs to one controller, store the values in a variable that the class has access too, and then call the main controller with another fetch that won't catch all the messages. I'll start to implement this now, but is that a good idea? Will it work?

Seems this question is more suitable for Stackoverflow, as it is related to Javascript and browsersupport more then to CI. I have read original thread and this one, and to be honest I doubt EventSource is a best approach based on code you are showing, and concidering that none of the browsers support it completely (https://caniuse.com/#feat=streams). Probably batch processing would be easier to implement and more stable.

I doubt EventSource is a best approach based on code you are showing, and concidering that none of the browsers support it completely (https://caniuse.com/#feat=streams). Probably batch processing would be easier to implement and more stable.

Isn't this the relevant browser support page?

Oops, seems I misunderstood. Apologies. Disregard my reply please.

Oops, seems I misunderstood. Apologies. Disregard my reply please.

Well, in any case, what are my options? Would websockets work? and if so, how do I implement it? I just downloaded Ratchet.

---

Seems this question is more suitable for Stackoverflow, as it is related to Javascript and browsersupport more then to CI. I have read original thread and this one, and to be honest I doubt EventSource is a best approach based on code you are showing, and concidering that none of the browsers support it completely (https://caniuse.com/#feat=streams). Probably batch processing would be easier to implement and more stable.

Also, I already asked stack here, but ,y question seems to have no bites...

At this point, I just want to send updates to the user on the progress of this process/data conversion. I need what ever tech we use to work on a AWS.

Hey, I already said this, but thanks for all the help! Do you guys have a patron by any chance?

Well, after I was ashamed with my quick and unprofessional response into this thread I could not get asleep without  putting my hands on this.

Few interesting learnings, probably would be useful for someone:

1. Code from above did not work for me. I observed weird behaviour in all browsers: after receiving exactly 3 messages from the server, my browser disconnected and attempted to reconnect. It was an endless loop.
   
2. It turned out that my web-server was running with ob_get_level() == 2. I could get rid of the issue only after I recursively flushed all buffers in the CI controller.
   
3. Finally, I had a problem with saying both server and client to stop working. Client was complaining on suddenly appearing wrong MIME not equal to expected 'text/event-stream', while client always wanted to restart listening to the stream, and could not understand that server is done with messaging. 
   

So here is my code that works well both on server and client side:

Client (trivial wrapping omitted):

Code:

<button onclick="sse()">Start SSE</button>

<script>
    function sse() {
      let eventSource;
      eventSource = new EventSource('<?= route_to("stream")?>');

      eventSource.onopen = function(e) {
        console.log("eventSource: connection opened");
      };

      eventSource.onerror = function(e) {
        console.log("eventSource: error occurred");
        if (this.readyState == EventSource.CONNECTING) {
          console.log(`eventSource: reconnecting (readyState=${this.readyState})...`);
        } else {
          console.log("eventSource: connection failed");
        }
      };

      eventSource.onmessage = function(e) {
        console.log("eventSource: " + e.data);
        if (e.data == 'done') {
          this.close();
          console.log("eventSource: server is done");
        }
      };
    }
</script>

And method in CI Controller that is called from above via named route for simplicity:

PHP Code:

<?php namespace App\Controllers;
use CodeIgniter\I18n\Time;

class Streamer extends BaseController {

    public function stream() {
        header("Content-Type: text/event-stream");
        header("Cache-Control: no-cache");
        
        // My server had 2 output buffers already opened before it reached here
        // one was "default output handler" from php.ini
        // another was "Closure::__invoke" I have no clue where from
        // here might be even more levels if gzip is enforced for example
        while (ob_get_level() > 0) {
            ob_end_flush();
        }

        // Here will be data sourcing from DB and processing by external API
        for ($i=0; $i < 10 ; $i++) {
            echo "data:", "Server time is ".date(DATE_ISO8601)."\n", "id: $i\n\n";
            flush();
            sleep(1);
        }
        echo "data: done\n\n"; // Gives browser a signal to close connection
        flush();
        exit; // without it was throwing "Headers already sent" warning
    }
} 


This works like a charm in Safari, Chrome and Firefox (all desktops). No need for websockets.

Useful articles I found on this topic: 

• https://www.html5rocks.com/en/tutorials/...ce/basics/
  
• https://javascript.info/server-sent-events
  
• https://gist.github.com/Deele/ebeb6b9047bab0939e78
  

Thanks once again for starting this thread!

Probably, CI4.x could include some out of the box support for streaming, what do you think?

Well, after I was ashamed with my quick and unprofessional response into this thread I could not get asleep without  putting my hands on this.

Few interesting learnings, probably would be useful for someone:

1. Code from above did not work for me. I observed weird behaviour in all browsers: after receiving exactly 3 messages from the server, my browser disconnected and attempted to reconnect. It was an endless loop.
   
2. It turned out that my web-server was running with ob_get_level() == 2. I could get rid of the issue only after I recursively flushed all buffers in the CI controller.
   
3. Finally, I had a problem with saying both server and client to stop working. Client was complaining on suddenly appearing wrong MIME not equal to expected 'text/event-stream', while client always wanted to restart listening to the stream, and could not understand that server is done with messaging. 
   

So here is my code that works well both on server and client side:

Client (trivial wrapping omitted):

Code:

<button onclick="sse()">Start SSE</button>

<script>
    function sse() {
      let eventSource;
      eventSource = new EventSource('<?= route_to("stream")?>');

      eventSource.onopen = function(e) {
        console.log("eventSource: connection opened");
      };

      eventSource.onerror = function(e) {
        console.log("eventSource: error occurred");
        if (this.readyState == EventSource.CONNECTING) {
          console.log(`eventSource: reconnecting (readyState=${this.readyState})...`);
        } else {
          console.log("eventSource: connection failed");
        }
      };

      eventSource.onmessage = function(e) {
        console.log("eventSource: " + e.data);
        if (e.data == 'done') {
          this.close();
          console.log("eventSource: server is done");
        }
      };
    }
</script>

And method in CI Controller that is called from above via named route for simplicity:

PHP Code:

<?php namespace App\Controllers;
use CodeIgniter\I18n\Time;

class Streamer extends BaseController {

    public function stream() {
        header("Content-Type: text/event-stream");
        header("Cache-Control: no-cache");
        
        // My server had 2 output buffers already opened before it reached here
        // one was "default output handler" from php.ini
        // another was "Closure::__invoke" I have no clue where from
        // here might be even more levels if gzip is enforced for example
        while (ob_get_level() > 0) {
            ob_end_flush();
        }

        // Here will be data sourcing from DB and processing by external API
        for ($i=0; $i < 10 ; $i++) {
            echo "data:", "Server time is ".date(DATE_ISO8601)."\n", "id: $i\n\n";
            flush();
            sleep(1);
        }
        echo "data: done\n\n"; // Gives browser a signal to close connection
        flush();
        exit; // without it was throwing "Headers already sent" warning
    }
} 


This works like a charm in Safari, Chrome and Firefox (all desktops). No need for websockets.

Useful articles I found on this topic: 

• https://www.html5rocks.com/en/tutorials/...ce/basics/
  
• https://javascript.info/server-sent-events
  
• https://gist.github.com/Deele/ebeb6b9047bab0939e78
  

Thanks once again for starting this thread!

Probably, CI4.x could include some out of the box support for streaming, what do you think?

Thanks for the help, zahhar! I don't see much that is different, but will this work while the first http request id processing? What is happenning is that we send the form data and want the receive updates on that data's processing, but we can't because the browser is waiting for a reply to the form.