Tag Archives: Infrastructure

Report of visit at datacenter on 2021-04-19

After a new member was added to the access engineers team it was possible to visit the datacenter following the epidemiological guidelines for SARS-CoV-2, as well as our own security guidelines.

During this visit we applied the long-awaited patch for bug 1438 by adding the serial number to certificate revocation lists.

This visit also provided an opportunity to instal a new infrastructure-server, courtesy of Abil’I.T. , a Luxembourg based free software service provider. Many thanks again!

… and …

We did the Class-3-resigning during this visit. Currently we’re testing this new Class-3-certificate and will publish it real soon.

A new visit in the summer will be necessary to replace hardware (and maybe apply further patches on the signer).

L’infrastructure de CAcert se prépare pour le futur

Samedi passé, le 13 juillet 2019, dans une opération coordonnée, les équipes Infrastructure et Interventions Critiques de CAcert ont mis à jour avec succès le système d’opération de notre infrastructure, localisée en Hollande. Le système repose désormais sur la version Buster de Debian, version déclarée stable la semaine passée par le projet Debian.

[this blog post is in French – for English, see here]

Déroulement de l’opération
Les équipes ont démarré leur travail ce matin vers 9:30 CEST et achevé toutes les mises à jour à 16:30 HAEC. Certaines de nos applications n’ont été rétablies qu’un peu plus tard. L’ensemble de nos systèmes est à présent revenu à son fonctionnement optimal.

Quelles améliorations en découlent?
La mise en service d’une version moderne du système d’exploitation, dont dépend toute notre infrastructure, garantit pour le futur un fonctionnement meilleur de nos applications:

  • La technologie de containers LXC est ainsi passée de la version quelque peu anti-diluvienne 0.8.0 pre-release à la version 3.0.3, qui offre désormais une API bien définie, une sécurité améliorée, et qui sera beaucoup plus facile à manipuler pour les administrateurs de nos applications.
  • Le pare-feu et la translation d’adresses devraient avoir été accélérés, par rapport à ce que parvenait à faire l’ancienne configuration iptables. Nous continuons à utiliser ferm comme wrapper, mais l’équipe Infrastructure de CAcert envisage déjà son remplacement par des règles nftables natives, qui offriront une analyse du trafic aussi sûre mais plus rapide.
  • On peut retrouver sur notre liste de discussion d’autres détails de cette importante mise à jour.

Le leader de l’équipe Infrastructure de CAcert, JanDD, a fait part de sa grande satisfaction d’avoir vu s’accomplir enfin cette mise à niveau cruciale, offrant une solidité accrue de nos services aux membres de notre communauté. Dans un commentaire publié en fin d’après-midi, il remerciait chaleureusement Wytze, leader de l’équipe des Interventions Critiques, pour l’apport de sa compétence tout au long de la journée.

Les bénévoles dont sont formées ces deux équipes ont ainsi délivré un travail en continu pendant 7 heures et demi aujourd’hui samedi, pour pérenniser nos systèmes. Joignez-vous à nous pour les en remercier et marquez votre soutien par un don à l’association [x]. Les dons collectés sont exclusivement destinés à couvrir le coût de notre infrastructure (hébergement des serveurs sur site sécurisé et consommation électrique). “Par ma contribution financière, j’encourage le travail de Jan et Wytze; c’est ma façon de leur dire merci”.

Si vous observiez un dysfonctionnement consécutif à cette récente mise-à-jour, n’hésitez pas à nous en faire part sur https://bugs.cacert.org/ (suivre les onglets “Infrastructure” >
“Infrastructure hosts”).

Si vous cherchez à rejoindre l’une de nos équipes techniques, abonnez-vous à la liste de discussion de nos développeurs, ou contactez directement notre secrétaire.

CAcert’s infrastructure ready for the future

On saturday, 13th of July 2019, in a joint operation, CAcert Infrastructure Team and CAcert Critical Team updated the operation system of CAcert’s infrastructure in the Netherlands sucessfully. The system is now running on the Debian Buster OS release that has been released by the Debian project last weekend.

Timing

The teams started this morning at around 9:30 CEST and finished the upgrades at 16:30 CEST, some of our applications turned back to service afterwards. The system is running smoothly now.

What is new?

The new OS release provides some features that are important for our infrastructure and will allow better operation of our applications in the future:

  • LXC has been upgraded from the somewhat primitive 0.8.0 pre-release to LXC 3.0.3 that has a proper API, better security and which will help application administrators
  • Firewalling/forwarding/NAT should now be faster then the old iptables setup. We still use ferm as a wrapper but the CAcert Infrastructure Team is already considering switching to native nftables rules that will provide a similar but faster rule set.
  • Further details about this major update can be read on our mailing list.

CAcert Infrastructure Team Lead JanDD is happy that we could finish this big upgrade and that we could implement all these changes for you. In a statement made on the early saturday evening, he thanked again to Wytze from CAcert Critical Team for his great support during the day.

The volunteers from these two teams worked for seven and a half hours today, Saturday, to keep our systems up to date. Join us in thanking them and donate now at your own discretion. Your donation will only be used to pay for the infrastructure (hosting, electricity in the data center). «I say thank you to Jan and Wytze and their team with a donation!»

If you find any issues that might be caused by the upgrade feel free to file bugs on https://bugs.cacert.org/ (at project Infrastructure > Infrastructue hosts).

If you want to join one of our teams, please join the development mailing list or write to the secretary.

Service interruption for a major system upgrade for a brighter future of CAcert on Sat, 13-07-2019

The CAcert Infrastructure Team will perform a major system upgrade of our infrastructure host tomorrow, 13th of July 2019, starting at 8am UTC/10am CEST. Wytze van der Raay of the critical infrastructure team will assist via remote console if necessary.

We expect the upgrade to run for at least 4 hours and some services might need fixes that will require even longer.

Most services will be unavailable at least for parts of the upgrade session. We will try to keep the downtime of essential services (email, emailout, lists, blog, wiki) as short as possible. We hope to not cause to many inconvenience but we cannot wait longer to perform these long needed update. The Debian Buster stable release last week and the recently acquired knowledge on how to use the remote console system of infra02 inspired us to perform the upgrade now.

Jan Dittberner
CAcert Infrastructure Team Lead

Please support the hudge work of the volunteers of our Infrastrucutre Team, please donate to continue to run this service. Thank you.

Stability of e-mail verification strongly improved

The e-mail verification on the CAcert web server has recently led to repeated support requests. An analysis of the log files in our data center showed that the corresponding error occurred more frequently. So we have to conclude that many CAcert users have been negatively affected. In order to avoid further negative effects, an emergency
patch was deemed necessary by the Critical System Administrator Team.

The standardised review and testing of the emergency patch implemented yesterday is carried out by the regular teams in the aftermath, which can result in a formal blessing for this patch or a request for additional code or configuration changes. We would like to thank the Critical System Administrator Team for their quick and decisive action. All teams consist of volunteers. If you want to support the work done by the Critical System Administration Team and the review by the Software Team, please donate, to continue to run this service. Thank you.

Updated: Information about Heartbleed-bug in OpenSSL 1.0.1 up to 1.0.1f

German version below

There is news about a bug in OpenSSL that may allow an attacker to leak arbitrary information from any process using OpenSSL.

Good news:

Certificates issued by CAcert are not broken and our central systems did not leak your keys.

Bad news:

Even then you may be affected.
Although your keys were not leaked by CAcert your keys on your own infrastructure systems might have been compromised if you were or are running a vulnerable version of OpenSSL.

To elaborate on this:

The central systems of CAcert and our root certificates are not affected by this issue. Regrettably some of our infrastructure systems were affected by the bug. We are working to fix them and already completed work for the most critical ones. If you logged into those systems, within the last two years, (see list below) you might be affected!
But unfortunately given the nature of this bug we have to assume that the certificates of our members may be affected, if they were used in an environment with a publically accessable OpenSSL connection (e.g. Apache webserver, mail server, Jabber-Server, …). The bug has been open in OpenSSL for two years – from December 2011 and was introduced in stable releases starting with OpenSSL 1.0.1.
When an attacker can reach a vulnerable service he can abuse the TLS heartbeat extension to retrieve arbitrary chunks of memory by exploiting a missing bounds check. This can lead to disclosure of your private keys, resident session keys and other key material as well as all volatile memory contents of the server process like passwords, transmitted user data (e.g. web content) as well as other potentially confidential information.
Exploiting this bug does not leave any noticeable traces, thus for any system which is (or has been) running a vulnerable version of OpenSSL you must assume that at least your used server keys are compromised and therefore must be replaced by newly generated ones. Simply renewing existing certificates is not sufficient! – Please generate NEW keys with at least 2048 bit RSA or stronger!
As mentioned above this bug can be used to leak passwords and thus you should consider changing your login credentials to potentially compromised systems as well as any other system where those credentials might have been used as soon as possible.
An (incomplete) list of commonly used software which include or link to OpenSSL can be found at related apps.

What to do?

  • First ensure that you upgrade your system to a fixed OpenSSL version (1.0.1g or above).
  • Only then create new keys for your certificates.
  • Check what services you have used that may have been affected within the last two years.
  • Wait until you think that those environments got fixed.
  • Then (and only then) change your credentials for those services. If you do it too early, i.e. before the sites got fixed, your data may be leaked, again. So be careful when you do this.

What we are doing:

  • We are updating the affected infrastructure systems and and create new certificates for them.
  • We use this opportunity to upgrade to 4096 bit RSA keys signed with SHA-512. The new fingerprints can be found below. 😉
  • We will contact all members, who had active server certificates within the last two years.
  • We will keep you updated, here.

Press release

CAcert-PM-Heartbleed-en

Update:

There is a website where one can check if a domain is affected:
http://filippo.io/Heartbleed/ (Use on your own risk)
We checked our systems against it, it looks like most of the systems we classified as affected are not actually affected. But we decided to update them, anyway and provide them with new certificates, as well.

Status of CAcert Infrastructure systems:

Not affected / Nicht betroffen:

  • main website (www.cacert.org)
  • certificate signer (the system doing the actual certificate issuing)
  • email system (inbound and outbound mail servers)

Fixed/Behoben:

blog.cacert.org
SHA1 Fingerprint=7A:13:19:98:1E:FB:9F:F9:9A:E6:3D:E5:7D:F0:42:E1:BE:56:B9:79

board.cacert.org
SHA1 Fingerprint=9B:30:44:3D:A8:8C:F5:5E:50:07:68:70:D6:A1:83:44:F9:7D:00:D6

bugs.cacert.org
SHA1 Fingerprint=E1:2C:12:7F:66:DF:2E:9D:F6:BC:FB:6F:BC:F1:2E:A0:10:5F:8E:BA

cats.cacert.org
SHA1 Fingerprint=2E:0B:08:6E:F1:48:FB:76:69:D6:6D:51:8F:E9:B3:2C:C3:4B:14:25

community.cacert.org
SHA1 Fingerprint=9B:8E:0A:68:96:E5:C8:E6:E6:8E:D8:10:31:3F:7C:2C:A8:4E:E1:3F

email.cacert.org
SHA1 Fingerprint=27:AB:9B:90:51:9B:BA:60:51:B3:84:54:FF:C7:09:94:63:86:68:FA

git.cacert.org
SHA1 Fingerprint=87:47:41:6D:C1:32:C7:22:00:E5:DA:E5:3C:4B:28:A2:2B:8A:F3:E4

irc.cacert.org
SHA1 Fingerprint=57:F4:0F:38:1E:53:D0:83:DC:D2:40:0A:13:98:B7:06:55:EA:A7:19

issue.cacert.org
SHA1 Fingerprint=82:33:D3:AE:32:56:C5:AD:9E:BF:D1:84:62:56:EA:95:31:7E:64:8C

lists.cacert.org
SHA1 Fingerprint=6A:AE:16:90:A2:1F:CC:1B:B7:93:71:C0:1B:BD:2E:14:68:69:45:EA

svn.cacert.org
SHA1 Fingerprint=D5:00:0A:15:17:04:2F:50:5B:09:3C:DD:B9:0A:57:DD:B3:BE:3D:B4

translations.cacert.org
SHA1 Fingerprint=B7:59:B9:BA:46:64:E2:D4:C8:73:20:50:45:9B:08:5E:2B:DF:D0:1B

wiki.cacert.org
SHA1 Fingerprint=87:07:59:30:30:64:27:15:6E:39:C3:66:09:CA:7A:90:7D:2F:32:32

cacert.eu
SHA1 Fingerprint=A2:7A:CB:E7:91:0A:ED:7E:63:9F:D1:97:01:96:E9:7B:F0:9E:43:3D

Affected/Betroffen:

  • Testserver-management-system (you should have not used correct data there)

Deutsche Fassung:
Ein Bug in OpenSSL wurde gefunden, der es einem Angreifer erlaubt beliebige Informationen jedes Prozesses zu erlangen, der OpenSSL nutzt.

Die gute Nachricht:

Die von CAcert ausgestellten Zertifikate sind nicht kaputt und unsere zentralen Systeme waren auch nicht angreifbar und hat keine Schlüssel verraten.

Die schlechte Nachricht:

Dennoch kann jeder betroffen sein!

Um ins Detail zu gehen:

Die zentralen Systeme und die Stammzertifikate von CAcert sind von diesem Problem nicht betroffen. Leider sind einige unserer Infrastruktur-Systeme durch den Fehler betroffen.
Wir arbeiten daran diese zu fixen und haben dies auch schon für die meisten erledigt. Jeder, der sich auf diese Systeme in den letzten zwei Jahre eingelogt hat kann betroffen sein!
Aufgrund der Art des Fehlers, müssen wir leider davon ausgehen, dass die Zertifikate unserer Mitglieder betroffen sind, wenn sie sich in eine Umgebung eingelogt haben, die über öffentliche OpenSSL-Verbindungen zugänglich war (z.B. Apache Webserver, Mail Server, Jaber-Server, …). Dieser Fehler war zwei Jahre lang in OpenSSL – seit Dezember 2011 – und kam beginnend mit Version 1.0.1 in die Stabilen Versionen.
Angreifer, die einen verwundbaren Service erreichen, können die TLS-Erweiterung “heartbeat” ausnutzen, um beliebige Abschnitte aus dem Speicher zu erlangen, indem sie eine fehlende Bereichsprüfung ausnutzen. Das kann zur Offenlegung von privaten Schlüsseln, im Speicher abgelegte Sitzungsschlüsseln, sonstige Schlüssel genauso wie jeglicher weiterer Speicherinhalt des Server-Prozesses wie Passwörter oder übermittelte Benutzerdaten (z.B. Webinhalte) oder andere vertrauliche Informationen führen.
Die Ausnutzung dieses Fehlers hinterlässt keine merklichen Spuren. Daher muss für jedes System, auf dem eine angreifbare Version von OpenSSL läuft (oder lief), angenommen werden, dass zumindest die verwendeten Server-Zertifikate kompromittiert sind und deswegen durch einen NEU generierte erstetzt werden müssen. Einfach die alten Zertifikate zu erneuern, reicht nicht aus! – Bitte NEUE Schlüssel mit 2048 Bit RSA oder stärker generieren!
Wie oben erwähnt kann dieser Fehler ausgenutzt werden, um Passwörter zu entwenden. Daher sollte jeder überlegen, alle Zugangsdaten zu möglicherweise betroffenen Systemen und allen Systemen bei denen diese sonst noch verwendet worden sein können, so bald wie möglich auszutauschen.
Eine (unvollständige) Liste an weit verbreiteter Software die OpenSSL verwendet kann z.B. unter folgendem Link gefunden werden.

Was ist zu tun?

  • Als erstes müssen die eigenen Systeme auf eine fehlerbereinigte Version von OpenSSL aktualisiert werden (Version 1.0.1g oder neuer).
  • Danach neue Schlüssel für die Zertifikate erstellen. Jetzt ist es sicher das zu tun.
  • Überprüfen, welche fremden Dienste in den letzten zwei Jahren besucht worden sind.
  • Warten, bis dort wahrscheinlich der Fehler behoben wurde.
  • Dann (und erst dann) die Logindaten für diese Dienste erneuern. Vorsicht: Wenn das zu früh getan wird, also wenn der Dienst noch nicht bereinigt wurde, können die Daten wieder abgegriffen werden.

Pressemitteilung

CAcert-PM-Heartbleed-de

Update:

Es gibt eine Webseite, bei der man seit kurzem checken kann, ob eine Domäne angreifbar sind:
http://filippo.io/Heartbleed/ (Benutzung auf eigene Gefahr)
Wir haben unsere Systeme dagegen geprüft und es sieht so aus, als wären nicht alle, die wir als potentiell angreifbar eingestuft haben tatsächlich angreifbar, aber wir haben uns dennoch dafür entschieden die entsprechenden Systeme aufzurüsten und die Zertifikate zu erneuern.

Was wir tun:

  • Wir arbeiten daran, alle Infrastruktur-Systeme auf den neuesten OpenSSL-Stand zu bringen und für diese neue Zertifikate zu generieren.
  • Wir nutzen diese Gelegenheit, um dabei auf 4096er-Schlüssel, die mit SHA-512 signiert sind aufzurüsten. Die neuen Fingerabdrücke können oben gefunden werden. 😉
  • Wir werden alle Mitglieder kontaktieren, die in den letzten zwei Jahren aktive Server-Zertifikate benutzt haben.
  • Wir werden neue Informationen an dieser Stelle veröffentlichen.

CAcert server move completed on December 12, 2013

The move of all existing CAcert servers  to a new smaller rack at the current hosting centre has completed, mostly successful, on December 12 at 23:00 UTC. All main services are available again now, but we still have some smaller problems to sort out, mostly due to the switch-over to a new much more compact firewall with a completely new architecture.

So please bear with us while we iron out the remaining problems, and feel free to report any issues you are still encountering.

Please join me in expressing thanks to the team that worked very hard over four hours after an already stressful day to get this major job completed: Stefan Kooman, Mendel Mobach, Martin Simons.

Blog update

The CAcert Blog (incl. its system) has been upgraded to recent software versions. With this also a new Theme had to be created to be compatible with the current version of the blog software.

Please let me know, if anything was broken by the upgrade or anything does not work for you as expected.

Maintenance / Changes infrastructure

[Update: finished.] I will begin with the work now 2012-04-01 00:00 UTC. Infrastructure e.g. email, wiki, blog, lists will be unavailable during the time. The CAcert website along with all critical systems are not “infrastructure” and will not be affected.

I am planning to work on infrastructure again this weekend (2012-03-31) and downtimes may occur. Our host is low on disk space so I need to do some restructuring. A larger downtime may be involved.

Currently, each virtual host has its own logical volume (LVM). They will be consolidated into one. During the move and resizing I think I will not get around shutting down all infrastructure services.

I will send out a notice before I begin the work.
Sorry for any inconvenience.

—-

I am currently continuing the work until 2012-03-26 07:00 UTC. The following services may be effected (the other services have been migrated already):

  • email
  • lists
  • emailout
  • board
  • translingo

I am working on infrastructure systems. During the next 6 hours the following services may be down as they are moved to another server:

* bugs
* irc
* webmail
* wiki
* email
* cats
* lists
* issue
* emailout

The following services have been shut down:

* dupes
* forum
* logging
* paypal
* test2