I guess for cleanliness this should also remove the RPM package from /tmp? I think I missed that in the WaiverDB one too.

So... who or what is filling in the variables here like ${DATABASE_PASSWORD}? Is that a feature of the config parser that resultsdb is using? Because OpenShift itself has no templating functionality for config files, right? That was why I had to add support in WaiverDB for passing in the db password and Flask secret key through the environment instead of the config file.

Does resultsdb care what user it runs as? Does it actually require a user named resultsdb to exist? Does it write anything to the filesystem ever? Surely it doesn't?

I personally think this nss_wrapper approach is a lot hackier and messier than just telling the container to run as an arbtirary user id (as in USER 1001).

I think the right way to do this is to have a separate container/pod specifically for populating or upgrading the database, and have OpenShift fire that off as part of each new deployment. The problem with doing it here inside the app itself is that it's racey (you will be starting multiple copies of the app and each one will be trying to do the init_db). WaiverDB makes this same mistake right now and we need to fix it in there too.

Anyway -- not necessarily something that you need to fix now. This should be okay as a starting point.

Right, I'll add this.

It's oc process doing the magic. If you run something like:

oc process -f openshift/resultsdb-test-template.yaml -p TEST_ID=123 -p RESULTSDB_APP_VERSION=latest --local -o yaml

you can check that all these are nicely replaced with the template parameters.

(Actually I did not know about this either, I just assumed that works this way :) )

What about this? This expects the image to be pushed in the internal registry of the cluster. I would say this is more portable than referencing an internal registry, but it hard-codes the registry IP and project name - I should check if there are some built-in template variables that could be used for these.

I've used the nss_wrapper approach b/c this was the recommended way in the docs, although I do agree that it's hackish and messy.
Besides this, I cannot think of a good reason why the 'USER 1001' approach wouldn't work in this case (maybe it will be less future-proof), so I'll make this change.

I think the right way to do this is to have a separate container/pod specifically for populating or upgrading the database, and have OpenShift fire that off as part of each new deployment.

Wouldn't this stop us from doing rolling releases? Shouldn't resultsdb to be able to handle a situation when the database is updated, but there are still some pods running older versions?

You are right about the racey aspect, I'll have to search for some ideas on how this could be handled.

So this means that the container will end up running the app inside a Flask development server, which is explicitly not designed for production use:

http://flask.pocoo.org/docs/0.12/api/#flask.Flask.run

"Do not use run() in a production setting. It is not intended to meet security and performance requirements for a production server. Instead, see Deployment Options for WSGI server recommendations."

So this is okay for now, if this container is only intended for test environments. But if we want to run the container in production it will need to use a proper server, like gunicorn.

Oh yeah I see. So that means this won't actually be usable as is for a "real" deployment (prod or prod-like) since it would mean the secret is embedded directly inside the config map. It should be passed in as an env var to the container instead, which might mean patching ResultsDB to accept its secrets from environment variables the way I had to do for WaiverDB as well.

But I guess this is okay as a first step, just for test environments.

Just bear in mind that the so-called Open Platform we have access to inside of Red Hat does not expose its internal registry. That is why for WaiverDB I am using that other, separate registry... Realistically there is no reason this couldn't just go to the public Docker Hub though. Just need to find (or register) a good namespace. Maybe there is already something that Fedora infra tools are published under...

Yeah it means for rolling deployments you actually have to design the db schema migrations to be always forwards- and backwards-compatible with whichever app versions still exist. That's how you achieve outage free updates... You have to first add *new* schema elements, apply that to the db, let the application code be updated, and then only after all old versions are no longer running you can *remove* old schema elements. It's more work but it's how you have write a so-called "cloud native" app. Right now I guess ResultsDB's migrations are not designed that way though. I'm not sure if there is any good answer.

Btw not sure if you already figured this out... the nss_wrapper stuff is mentioned in the OpenShift docs *in addition* to the USER directive. nss_wrapper is only needed in case the application needs to perform getent lookups of its own username (which a normal application will never need to do). But regardless whether you use nss_wrapper, you still need USER 1001 or any other unprivileged user id. OpenShift won't let the container run as root (which is the default if USER is absent).

OpenShift might have been letting you get away with this when you ran your own local cluster, but a real deployment (like the Open Platform internally inside Red Hat) doesn't permit it.

This is a bit unfortunate... I guess the problem is CentOS 7 mod_wsgi is too old to ship the mod_wsgi-express script? And even the Fedora rawhide package doesn't include it either, perhaps by mistake...

Could you file an RFE against the Fedora package, so that in future we could switch this to using packaged mod_wsgi instead of downloading stuff from PyPI?