Maintenance - 29-30th of September

cloud is back since yesterday

A bit more of background on the last events here https://forum.indiehosters.net/t/postmortem-cloud-migration/2915

@rainerzufall @jeremiejoudioux my apologies for the lack of information, there were a lot of unknown variables that I have to say lead to a bit of tensions :slight_smile:

We will improve this in the future, hopefully we should not get to such a critical situation anyways.

So far our main channel to reach all users is through nextcloud announcements but if its down that’s becoming an issue. The forum is probably the best way for such announcements.

thanks a lot for this update!

That’s the thing with communication. I was not sure whether it is really up, so I waited for your confirmation.
Interestingly I lost some of my one-time-passwords I have created for the Android Nextcloud app as well as Davdroid. Is that to be expected?

Happy to hear that as well!

unfortunately the lost of the password can be a side-effect

Thanks for the update and the more detailed insight linked above!
Do I understand it correcty that you transitioned all the data to the new infrastructure now? Or is there more to do?

Everything has been migrated to the new infrastructure, nothing more to do other than making sure everything is running fine there :slight_smile:

Hi there,
I am trying to update some scripts on my cloud and they get saved, but the updates don’t show on my webpage online…
Could you help me there?
Best,

Hi @mateo ,
we are going to fix this tomorrow see:
https://forum.indiehosters.net/t/maintenance-23rd-of-october-2018/2964

Ok, thanks, looking forward to it.
Best,
M

Are you still having troubles with your set up? Are you aware that over the last weeks at least once a day there are « Server Time Outs » and/or « Service Temporarily Unavailable »…

Is this already on your agenda also for the https://forum.indiehosters.net/t/maintenance-23rd-of-october-2018/2964?

@rainerzufall we are aware that there has been some up and downs but not that you get many timeouts.

Are you getting those timeouts when trying to login? Uploading big files or its random?

Yes, when trying to log in via a web browser. (but it should not be connected to my internet connection as everything else is working just fine.)

And my clients (desktop at work and also DAVdroid on my phone) do sometimes report connectivity issues saying that they can’t connect to my indie.host nextcloud (while my other two connected work nextclouds do work).

It is not connected to uploading (big) files.


If I can give more/better information, please just tell me how or what would be helpful.

Hi all

Just wanted to say that I also get very frequent issues since the maintenance, using the cloud apps through the browser (Firefox).

Sometimes, I get:

# 504 Gateway Time-out
nginx/1.13.12

Other times, I get:

# Not Found
The requested URL /apps/news/ was not found on this server.
Apache/2.4.25 (Debian) Server at cloud.indie.host Port 80

And other times, the page loads but I end up with only the top bar and nothing underneath.

me too… extremely slow for anything and sometimes time out.

@blub @stragu I know the cloud has been very slow lately. I’m still investigating on what is the bottleneck here… I will change a few things in the coming days to improve the performances on the php and mysql side, we will see if it gets better.

Plus I’d like to change the docker image we use, for the php-fpm one + nginx, but that will come a bit later.

Ultimately we will move the database to SSD disks but that comes with a cost that we need to think about.

A bit more background on what is going on:
So the cloud gets slower and slower as time goes on, at one point the health checks that we have setup restarts the cloud.

There was an issue, with the nextcloud apache config when the cloud restarts, leading to:

# Not Found
The requested URL /apps/news/ was not found on this server.
Apache/2.4.25 (Debian) Server at cloud.indie.host Port 80

Unfortunately I missed this a few times as it doesn’t appear in the monitoring, cloud restarts and you can login but you can’t get access to the pages.
This should not happen anymore.

Then when the cloud restarts, everyone is trying to reach the cloud in the same time, so yep timeouts.

Why is the cloud getting slower and slower, I’m still not sure. I changed a few things on the backup which could cause this issue, every 24h it is scanning and backuping millions of files.

2 « J'aime »

Sunday 04.11.2018:
I already noticed 6 times (14:09 / 15:22 / 21:18 / 21:32 / 21:47 / 22:00) a ‹ 502 Bad Gateway - The server returned an invalid or incomplete response. ›

Monday 5.11:
11:33
‹ 504 Gateway Time-out ›

Unfortunately I did not find time to look more into this issue.

We will spend some time this weekend with @pierreok to work on this. Obviously the migration did not go as planned and we are very sorry about all the inconvenience that it lead to. So Sunday we come back with some news and solutions (worst case would be to rollback to our previous setup).

As I already mentioned we are low on human resources at the moment, it has been a one man show for more than a year now. But we are at a stage that we can grow, an offer for to join IndieHost has just been published :). So things are going to be smoother in the coming months and I’ll be able to split the workload and focus the technical aspects. We are bit in a downtime before going a step further.

Again sorry about that. Sunday we come back with good news :wink:

1 « J'aime »

Is the issue with static websites supposed to be fixed? Mine is still not working (security issue with the https access to the website), although I can update files on the cloud.
Thanks,
M

Error message: « Este sitio usa Seguridad estricta de transporte de HTTP (HSTS) para especificar que Firefox sólo puede conectarse a él de modo seguro. Como resultado, no es posible añadir una excepción para este certificado. »