Forums

Hello, I am trying to develop a website which can display several line charts that are updated frequently (1 minute, 1 hour, 1 week). The task scheduler collects the data from multiple sources and saves them in a .csv file. Now, I'm stuck and not sure regarding the next course of action. Is it possble to achieve this ? What would be your suggestions ?

P.S. Website development rookie here.

Hmm, there are a couple of ways you could do that.

The most efficient way would be to have the tasks that are collecting the data (or perhaps other scheduled tasks) also generate static image files, which your website could serve. You could use matplotlib or something similar to generate line charts as images. The images would be saved in (say) /home/DC2/yoursite/static/images, and that directory would be made visible over the website's URL using the static files system (see my earlier link for details).

An alternative would be to generate the images on an as-requested basis -- that is, the code to generate the images would happen inside views in your website. This would be a little trickier -- you'd have to learn how to return non-text data from the views, which all frameworks support but it's a slightly less rookie-friendly thing :-) But it would have some advantages -- for example, it would make it easier to restrict access to the charts to people who were logged in, or to generate custom charts (different levels of zoom or something like that).

Well, I did try the first technique a few days ago, that is using the scheduler to also generate plots, which were then saved in the folder. However, that technique failed. Initially, I created a a static plot, which was rendered without any problem. The moment the plots were created and saved by the scheduler in the same folder as before, the plots were not being rendered & I could in no way identify the root of the problem. So, that made me question whether it is actually possible to do it via a scheduler. Another aspect is also the refresh rate of the webpage. I'm using Flask framework, so I do not know if that helps in this scenario.

The alternative technique (as-requested basis) is not preferred in this case as I would like to plots to continually update when the webpage is opened.

Any other suggestions/techniques or search directions :) ?

Hmm, just to make sure I make the right suggestions, could you give a bit more detail about what you mean by "I would like to plots to continually update when the webpage is opened"? Are you planning to have it dynamically changing without the user having to refresh the page?

Yes, the plan/intention is that once the user opens the website, the plots are changing continuously (approx. every minute) without the user having to refresh the page.

Thanks! OK, so I think the scheduled task solution is a good one, but you'll also need to have some JavaScript on your page to periodically fetch the images from the server so that they'll update.

Probably the first thing to do is to get the scheduled tasks working and outputting images so that they can be viewed when the user first visits the page, and then to sort out the JS stuff later. Could you give a little more detail on what was going wrong when you tried that? You mentioned that the plots weren't being rendered -- what was being displayed instead? Was it the normal "image not found" icon?

Hello, I had modified the code to such an extent that it took quite while to revert back to the original version, but I have not been able to replicate the problem. However, I have been able to solve this issue and am able to now display the image on the screen. :)

The current issue is with the always-on task scheduler. I noticed that the task switches off for roughly 32 minutes for every 4 hours of run. I do not get a message stating that it was either killed or stopped. This results in the previously stored data in a csv file being overwritten and no images being created during this span of time. So, I was wondering if there is any issue with the scheduler or something else that I need to account for while deploying a script on the scheduler.

hi- how are you noticing the 30min task switch off? what was the most recent switch off? maybe we can dig into log files to see if there was anything unusual then.

Through the log of the scheduled tasks as well the overwrite of csv file, I noticed the ~32 minutes of inactivity, the latest being 5.47AM, 29/3/19. This event has occurred over the past 2 days, but not after the mentioned event. But now, there was a new event, again with no process being killed, but resulting in overwriting of previous data.

Log (Only generated for this event, the others did not even have this)

Mar 29 15:24:13 Traceback (most recent call last):
Mar 29 15:24:13 File "/home/user/scripts/***trial_V3.py", line 67, in <module>
Mar 29 15:24:13 with urlopen ('http://blahblah.com/json_feed/123456789') as response:
Mar 29 15:24:13 File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
Mar 29 15:24:13 return opener.open(url, data, timeout)
Mar 29 15:24:13 File "/usr/lib/python3.7/urllib/request.py", line 525, in open
Mar 29 15:24:13 response = self._open(req, data)
Mar 29 15:24:13 File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
Mar 29 15:24:13 '_open', req)
Mar 29 15:24:13 File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
Mar 29 15:24:13 result = func(*args)
Mar 29 15:24:13 File "/usr/lib/python3.7/urllib/request.py", line 1345, in http_open
Mar 29 15:24:13 return self.do_open(http.client.HTTPConnection, req)
Mar 29 15:24:13 File "/usr/lib/python3.7/urllib/request.py", line 1320, in do_open
Mar 29 15:24:13 r = h.getresponse()
Mar 29 15:24:13 File "/usr/lib/python3.7/http/client.py", line 1321, in getresponse
Mar 29 15:24:13 response.begin()
Mar 29 15:24:13 File "/usr/lib/python3.7/http/client.py", line 296, in begin
Mar 29 15:24:13    version, status, reason = self._read_status()
Mar 29 15:24:13 File "/usr/lib/python3.7/http/client.py", line 265, in _read_status
Mar 29 15:24:13 raise RemoteDisconnected("Remote end closed connection without"
Mar 29 15:24:13 http.client.RemoteDisconnected: Remote end closed connection without response

Any leads/suggestions would be highly appreciated.

hmm- so this sounds like you were trying to make a url request and it failed in the middle- could it be that sometimes when you access that site, it hangs as a failure mode?

Based on the log, that appears to be the case in this instant. The other events did not generate this log, so I am not sure. I'll check for few more days to see if the event gets repeated.

Ok. Let us know what you learn and we'll see where this goes.