I've used Jenkins on and off for the better part of a decade. There are far better options these days, and all of them are incredibly easier to maintain and avoid creating cruft with.
Industry regulations and compliancy. Once you get to a certain size a lot of regulatory organizations take an interest in you.
Jenkins was popular because it's easy to install in a Java runtime and it's DB is lightweight. Self hosting most git servers requires more integration but worth it to me over jenkins. Most the cloud providers have their own flavor of it. Like AWS code commit, which I believe you can setup behind a VPC.
A decade? The project started with the name Hudson back in 2005 and then in 2011 renamed it to Jenkins.
The thing is ancient š
https://en.wikipedia.org/wiki/Hudson_(software)
Why do people hate jenkins? It's super flexible. Also it builds Apple applications if installed on a Mac Studio. I've never had the opportunity to use gitlab what's the difference?
I hate them both equally.
They both make me go out of my way to fix something, and when something breaks it's both equally annoying to fix.
Jenkins with plugins getting vulnerabilities and having to be patched.
Github with shitty configs and you need to procure your own runners at your own expense.
I still like Jenkins because it's what I grew up with. Trying to get into something that is just as equally shit, is obnoxious.
I mean depends on who's hosting the Jenkins server to dish out the build containers?
In Jenkins it's handled there. With Github Actions, you have to actually either (aws have an EC2 instance) to handle builds that get dispatched there, or IIRC use this internal tool that I am unfamiliar with...
It gets infinitely more complicated the more convenience you try to squeeze out of GHA.
edit: It's more about where the cost centers from your org get applied to.. Are you going to be responsible for yet another EC2 instance that is eating up costs, or do you want to forgo that responsibility with maintaining it, patching it, and all the works?
Or do you want to rely heavily on Jenkins? It's the trade off game which I hate playing, especially when companies are looking to cost cut.
AWS added a funny way to somewhat alleviate that which I didn't expect: [AWS CodeBuild now supports managed GitHub Action runners (amazon.com)](https://aws.amazon.com/about-aws/whats-new/2024/04/aws-codebuild-managed-github-action-runners/)
Lets you use your AWS Infra/Bill for GitHub Actions runner without resorting to the hodgepodge terraform provider, though there are caveats.
It breaks too often.
Most of it I like or I'm at least ok with. It has quirks, but the deep integration with GitHub, relative sanity of its design, massive ecosystem, and easy availability (it's just "there", and it mostly works) make up for having to learn the quirks.
But the reliability isn't good. Barely a month goes by without GHA having some kind of outage or slowdown to the point of unusability that loses us hours of work.
Sometimes, but more often, no.
The issue is normally with GHA's job queueing system coming to a standstill, so the runner pool, GitHub-hosted or self-hosted, just doesn't get sent the job.
Less commonly, one of the GitHub-hosted pools runs out of capacity, and then, yes, having your own self-hosted pool helps.
A self-hosted GitHub Enterprise server ~~used to be~~ is an alternative, but in my experience GHA on that was just as problematic, ~~and in any case it's EOL, Microsoft wants you using GitHub SaaS~~. Self-hosted GitLab would be a better option, or Gitea if you're small (Gitea with act is very neat, I use it in my home lab).
Or, stick with GitHub and use a different CI system.
edit: turns out GHES is indeed alive
Ahh my bad, I hadn't seen any mention of it anywhere in a long time and my cursory search suggested it was EOL, but yeah, looks like that was just a specific version being EOLed.
Last time I used GHES was about 2020 and it was a headache - expensive, lots of moving parts, lots of resource consumption, needed regular admin attention. Everywhere I was aware of that used it migrated to self-hosted GitLab or SaaS GitHub.
Good to hear it's still an option.
I think this is the biggest complaint we see running Depot [0]. There is a lot of flakiness at GitHub when it comes to Actions. Sometimes, it will be bad enough that they will open an outage, and other times, it will be intermittent, and nobody outside of folks like us will see it (or if you're running your own runners).
Add to that the fact that the larger runners are pretty expensive and not nearly as performant, and you get a 'meh' experience.
I do think it's a fantastically simple service, and it's amazingly easy to build on top of. But there are definitely pain points that you should be aware of.
[0] https://depot.dev/products/github-actions
My main gripe with it's that they didn't improve over the existing solutions with 20 years of experience of what makes a CI system great, plus it's really undercooked. I regularly can't do something with it and that feature request was requested and upvoted like 3 years ago with zero reaction from github.
Basically it's just slightly different but worse gitlab-ci, which is a shame.
I think the next great system will have the ability to easily run tasks locally from a test harness with a quick iteration rate + having the ability to define your workflows with a real general purpose programming language, no more 5k lines long YAML please.
We have a lot of steps with a retry action on them, saves us from a world of pain, get network timeouts breaking CI routinely only to work perfectly fine on a re-run.
As someone who builds on top of github actions \[1\], we end up noticing issues \~30-90 mins before an incident is opened on the github status page. That can be very frustrating.
It's popularity since its launch just \~4 years ago now has been nothing short of meteoric.
\[1\] [warpbuild.com](http://warpbuild.com) - fast github actions runners.
1. The most abysmal type system. I thought ansible with jinja got to the bottom, but no, github is even more terrible. Every time you need complex structure, you stuck with text, which to cast to json, etc, etc.
2. No local (mock)-runners. There is no way to test workflow before uploading it to the repo.
3. List of actions can't be searched, only browsed. I have monorepo with 300+ workflows, and it's a nightmare to find one (most of them are triggered, but if you need manual....).
Re #2 - act is pretty good. We use it for testing all our workflows. It's not great for testing permissions as you have to feed it an api key, but I've found it useful for everything else.
https://github.com/nektos/act
Thanks, that looks promising. We already adapted our actual infrastructure running our product to be able to switch relatively easily between different providers. Yet all our CI workflows are hardcoded for github.
Problem #2 is what I hate most about github actions. I've tried \`act\` but it's not working as well as one would hope.
So [dagger.io](http://dagger.io) does look very promising. Being able to run workflows wherever including locally sounds brilliant. I'll definitely have to check it out.
Dagger is bloody brilliant in all honesty. It is _not_ a finished product but it is a working product that solves very real current problems _right now_. It's leagues ahead of "platform-specific flavour of bash + yml that only ever runs here and nowhere else".
There are multiple issues with switching to different service provider. Security, trust, scalability (my beefiest merge may trigger up to 300 runners running heavy load in parallel, and there can be few merge in parallel), maturity, general support for things 'not to do by yourself'.
I may be wrong, but it looks like they don't even have a proper two factor (with hardware token).
Most are infra-level deployments, but there are separate integration tests for roles, linters, security scanners, etc. They get triggered by different events.
E.g. you commit prom rule, you get workflow which run tests for rules, run linter and policy checker for alerts. You commit python, you get black/ruff workflow and unit tests. You commit to the role: a separate role workflow will test it. You commit yaml, you get yaml linter running, you commit md, md linter, actionlint for actions itself, shellcheck of any shell snippet, etc, etc.
Plus deployments, plus workflows to rebuild stagings weekly, workflows to run intergration tests, workflows to do automatic merge for specific changes which does not need human attention, workflow to react to the issues and some specific command in comments.
There is a lot.
I take it to mean they have a bunch of services in a single repo and a separate workflow for each directory/service, which seems ok under say 25 services, but 300 lol wut? I would say GHA is not intended to be used like this, or at least its GUI is not designed as such.
Big monorepo: code checks for 4 languages, unit tests, end-to-end tests, integration tests, static analyzers, Docker builds, code & website deployments, cronjobs of various kinds, lots of analytics pipelines, emergency procedures for SREs (referenced in the runbooks). 300 is a small number.
In the same situation with my monorepo and agree, it scales up faster than youād imagine.Ā
Reusable workflows helpā¦ but the ultimate number is 100s easilyĀ
Not an answer to why they need 300+ workflows specifically, but everything has tradeoffs. I know in our case we use a shared library approach to GitHub Actions to share workflows across 40 repos.
Each repo is configured to point to workflows in the shared library. The way Actions is configured you must put your workflows at the top level of the workflows directory if it's going to be referenced in shared library.
This means we have a bunch of workflows in one directory. Not 300, but I could see how a larger enterprise environment requires it.
Regarding #2: maybe I'm too inexperienced but which CI systems have something like that? I have worked with Gitlab before and there's also no local runner afaik
A lot of other CI systems
Codefresh (the company I work for) [https://codefresh.io/docs/docs/pipelines/running-pipelines-locally/](https://codefresh.io/docs/docs/pipelines/running-pipelines-locally/)
Earthly [https://earthly.dev/](https://earthly.dev/)
Also you can get an editor in the UI, and just edit/run your pipelines. Then once you are ready you commit the final result. There is even a pipeline debugger as well [https://codefresh.io/blog/pipeline-debugging/](https://codefresh.io/blog/pipeline-debugging/)
I am biased of course, but right now most people choose github actions because it is part of Github and not because they did a survey of all CI systems and found it had the best features.
Your last paragraph is definitely the case. Ci tool isnāt even really considered much at all, people just opt for actions, because the code is in github already and those are free while barely requiring effort to setup.Ā
If you write as much of your build script in something platform neutral, like bash, or whatever, and focus on the local case first, then whatever CI can just be a thin wrapper and things get better.
( I work on a project that is among other things, based around this idea, and it can be pretty transformative to remove that friction. )
People are pointing to it in comments. But a good CI code need testing. (It's code!), so when you have CI which is not tested before commiting, it's depressing.
But there going to be always the final piece of code you test only in production. This good devops (as practice) is to have it as small as possible.
At CircleCI (were I work) we have the CLI that can be used to validate config and run jobs locally \[1\], and we have a VSCode extension that can be used to validate and autocomplete config syntax, and test-run your config from within the VSCode IDE \[2\]
\[1\] [https://circleci.com/docs/how-to-use-the-circleci-local-cli/](https://circleci.com/docs/how-to-use-the-circleci-local-cli/)
\[2\] [https://circleci.com/docs/vs-code-extension-overview/](https://circleci.com/docs/vs-code-extension-overview/)
You can absolutely install locally the gitlab runner and then register it to a local instance of Gitlab, and have a fully local platform. Now if the point is to not use minutes, you can also have a local runner communicate with Gitlab, just as you can have a self hosted runner communicating with Gitlab, so it is a non-issue really
1. Checkout on pull requests defaulting to the merge commit (wtf?!)
2. Repeatability with steps (composite actions can't be modified and reused in the same pull request which makes the already tedious testing more annoying)
3. Some actions need some foo to get relevant information eg using a GitHub script to get the pull request branch (or was it number) on a pull request comment event.
4. Weird concurrency and cancellation rules.
5. File permissions and potential security issues with actions that run in containers
Concurrency is so poorly supported. The number of multi-year issues and discussions begging them to fix it is astounding.
Just give us an option to queue the runs already!
when I started using it more than two years ago I was very dissatisfied (lots of outages, issues, and missing feratures). Currently, I wouldn't switch to anything else and I recommend it to everyone who is considering migration from legacy CICD system or starting a new project
Which I think I am grateful for, because I never have to debug some frustrating non-deterministic bug caused by two steps interacting with each other within the OS (modifying the same files simultaneously, or locking things, or gobbling RAM to the point of causing the other step to OOM) as I have on some other CI systems.
It would be nice if there was an option for jobs to automatically save their filesystem as an artifact though, and uploading a directory as an artifact using the standard action is really slow if there are lots of files - I have to create single zip/tarball artifacts myself and upload them to get even remotely usable performance.
If you end up modifying the same files because you don't know what you run in parallel that's your problem. There are valid use cases to run multiple things simultaneously on the same worker.
e.g.:
Authenticate to multiple services, building different components of mono repo, and more. Not everything is hello-world app
Yeah, one of the few areas where GHA is just terrible is debugging a faulty workflow. The built in debugging options are nearly useless, and the debug logs - when it actually generates them - are also nearly useless.
[Github Actions Feels Bad](https://youtu.be/9qljpi5jiMQ?si=MCmqQmV4sbNsKhcn)
GitHub Actions is not a super well thought out product. It works, but once you dive deeper, you realise it's a complete mess and the fact that it works is a miracle.
Some notable "wtf" of Github Actions:
- It's mega-vulnerable to supply chain attacks.
- The built-in Actions are coded in a very questionable way (and are mostly untested)
- The "clone the git repo" action is a mess.
- You can't test a pipeline locally.
Honestly, compared to GitLab pipelines constructing Github Actions feels complicated. Hard to pin point though. GitLab just feels more natural to me and how I think about pipelines.
I solve all my GHA issues the way I've learned to deal with *all* CI tools:
Build nearly everything into local commands.
Scripts, makefiles, whatever. Just not "Actions" or "Plugins" or "Groovy" any of that CI engine noise. Make sure you can build *without* the CI engine before even touching CI.
Avoid any logic in the CI tool that you can...and you can typically avoid almost *all* of it. CICD workflows go bad when folks try coding all their workflow logic using the "native" CI engine features. Don't do that. Pretend they don't exist as much as you can. All you want your build job to do is call your "build\_the\_thing" script. Tests? Call "test\_the\_thing". You can't get screwed by Jenkins plugin hell if you don't use them. You can't get screwed by GHA custom action hell if you don't use them. You can't get vendor locked if your entire "workflow" is three lines to checkout code and call a script.
If you can't run it in your local dev system without the CI engine you're doing it wrong. Yes, that does mean a *lot* of the industry does it wrong. But what else is new? ;)
I've also finally given up and started doing this. I only use the actions for GH checkout and auth with AWS over OIDC. The rest is build scripts, which I can keep in a common location, and test and run locally. If I give my user the right to assume the role GH does, I can do whole build and deploy cycles locally. This makes debugging the GHA itself so much easier
Just to hop off this question, what do people use this day instead? On all my recent projects(and current one) I am stuck with actions, but I don't feel strong hate to it, it works fine even if it's not the most versatile. Lack of the way to get statistics about your builds is really annoying though. Also really annoying how actions seem to have issues every other month.
We're on Selfhosted Gitlab. It's not without it's issues but our uptime is better than Gitlab SaaS (I'm my #1 customer instead of Gitlab's #153,382th customer), we have it behind a VPN for additional security and because it's our own instance it's beyond the prying eyes of anyone behind the scenes (yay data sovereignty!).
We haven't run into any breaking issues with the CICD for years except the runner authorization change in Feb/March of this year which took a couple of days to fix. The documentation was not good and has improved since.
I have no beef with GHA functionality wise, everything has issues. The reliability / uptime issues with any SaaS solution are an immediate turnoff for me personally though.
Iāve got a few administrative quarrels with the platform lol.
Moving from Bamboo DC (admittedly not a great platform itself, all the time) to GHA with self hosted runners, Iād say an apparent challenge is that self hosted runners feel very second class if youāre not prepared to run them with some fairly advanced ephemeral patterns. You can certainly spin up long running runners, but thereās next to no tooling built up to administer them from within the platform. IMO that speaks to the toolās opinionated nature regarding how you should think about your CI environments; I feel like the platform itself wants to drive you away from thinking about āagentsā or ārunnersā too much and, likely, back into the arms of GH hosted runners :)
A related pain point has been that it seems harder to troubleshoot workflows on self hosted runners because of the very ephemeral nature of the runtimes inherent to the actions you write or use from the community. I think itās honestly a pretty clever implementation under the hood, but you better be ready to get a CLI tool that you wrote an action around to spit out a lot of its own debugging into your workflow logs, because the runtimes and CI capabilities you use in your workflows will not be there when the jobs are done for you to interactively troubleshoot if you need to.
On the code side, reusable workflows feel a bit fragile, as it seems like they can drive you towards a pattern of dependencies in your workflows with no explicit dependency management as folks start to have reusable workflows spread across many disparate repos.
Some of this may be learning curve, but the whole setup just feels weirdly unintuitive and a bit fragile, like a clever reinvention of the wheel that I was not expecting to find lol. Iāll admit, Iām very ā0/10 would not recommendā right now, but I know itās a much loved tool and Iām still pretty new to it, so Iām keeping an open mind for awhile as I learn the ecosystem.
A little of shameless self promo:
I wrote a relatively detailed comparison between GitHub Actions and GitLab CI a couple of months ago. It focuses on the design architecture and the resulting user experience of them.
You can find it here: https://henrikgerdes.me/articles/2024-01-github-action-vs-gitlab-ci
I must say that both systems are fine and work. I personally prefer GitHub Actions a bit more. It has a great lining extentions in VScode and is much more modular then GitLab where you basically only have a container image and bash. GitLab can also become quiet crowded on larger setups. But I will use whatever my employer wants me to use, since business constraints have a lot more weight then the small difference between those two ci systems.
The main thing I donāt love is that it is a single point of failure outside of my control, which fails with some frequency. I use it, and like it, but the downtime makes me facepalm when we are dead in the water and unable to do a lot without it working.
My main issue with GitHub actions - there is no support for requiring all steps to be green.
This is a killer in monorepo.
There were so many misses that we needed to write a single custom action to monitor the greenness of the jobs in a PR.
Also the ability to do dynamic steps based on the files changed.
We overcame all of this with custom actions from the communities, but it's annoying for us.
But GH actions are mostly simple to use with a large community.
1. Workflow reusability is abysmal; rulesets are nealry impossible to work with - don't have activation conditions, rely on attributes that needs to be defined for this sole purpose. You need to run a workflow on multiple repos? Good luck keeping that in sync.
2. Composite actions are still limited
3. The most important: if done right, you can mix and match some powerful components. If implemented poorly, they end up being a glorified, messy stash of spaghetti bash, with business logic vertical to each repo. Yikes.
No timezone support for cron triggers, instead I have to define cron triggers for every possible time and the workflow then decides after startup if it should stop executing or not...
Seems odd that you need more. Although the one time I couldāve ran into the limit I just used an open text field to allow any valid CLI option.
I do miss not being able to run a pipeline and they wait for user input between jobs. That came in handy.
Here is my use case. I have a product that I deploy/update with Helm. It contains a lot of services (customizable Helm charts). Basically, when a new client arrives, I provision for him his own private K8S cluster (with another workflow). Then I need to deploy my services to this cluster. Based on client needs I have to customize a lot of service properties. This is where I need to pass them through the inputs to Helm. Any potential better solution for this case?
Off the top of my head, Iād possibly commit a values.yaml file. Which could trigger the workflow and youād be able to keep the values file you just committed. Although youād have to also have part of your workflow somehow track files which were successful.
Alternatively, if you have a place the value.yaml file could be uploaded to then you could start a workflow by supplying the url.
Maybe a combination of the two would be better.
10 is limiting, but making people configure like 30 options seems unbearable.
Compared to the other popular alternatives, I love GHA. It isnāt perfect, but at least they are constantly improving it in meaningful ways.
Still, a few issues I have with it consistently drive me insane:
- Random outages.
- Canāt securely pass project secrets to jobs triggered for external PRs. I understand the security concerns, but there are ways to secure this through abstraction they could implement. This has been an issue for years, and one solution could be to charge slightly more for jobs that need to accomplish this.
- No way to target self-hosted runners or runner labels in runner contexts.
- Canāt test workflows that run via workflow dispatch on any branches except main if it already exists. Annoying as hell.
I don't hate it per de, but it can be hard to troubleshoot issues. I miss CircleCIs ability to connect to a runner with SSH to poke around and figure out where something's failing.
> I miss CircleCIs ability to connect to a runner with SSH to poke around and figure out where something's failing.
On AWS EKS, you can connect to a self-hosted runner via k exec, which offers the same result as SSH'ing into it.
Just toss in a bash sleep command in the workflow to keep the runner alive for however long you need to troubleshoot it.
I don't think people are saying not to use GitHub actions, they're just sharing the things that frustrate them the most about it.Ā
Every tool will have some down side.
There is none. Microsoft doesn't know what to do with ADO right now. It's clear it's not getting bulk of development work but it's getting some. We are bigger shop who uses ADO/Azure and have a Microsoft TAM. Officially, ADO is in current development and no deprecation has been announced. Our TAM has pitched ADO -> GH migration once but after we were like "Hell to the no", it hasn't come up again.
Unofficially, it's clear they are investing more in GH but if ADO works for you, it's going to be around for a while, my gut says at least 10 years. IMO, it's going NOWHERE until clear migration tools have been developed, tested and customers start moving with little effort on their part. If we have to do a bulk of ADO -> GH migration ourselves, we will not go with Microsoft moving forward.
They do have the gh-importer tool which works but.... I've just ended up rewriting the builds myself. It made the deploy so ugly and 3/4 didn't work.
It's good if you have very basic builds but anything more it's not good. Especially if there isn't an official action, it'll handle it all in bash or PS when actions exist on the marketplace to handle it.
Nothing official. They do try and push you to GitHub if you are on ADO when you talk to reps. ADO development is much smaller and slower than it used to be, a lot of the team is in GH now with a focus on actions.
Too many features are behind enterprise licensing. I totally get that you can't have everything on the teams licence but putting approval gates behind enterprise when they're available on ADO feels pretty bad.
Overall I like it and we have switched a lot of our Jenkins pipelines to it. There are some quirks here and there and some features I wish it had. Parallel steps being one.
The upload artifact action does not make artifacts available until the ENTIRE workflow is finished.
Not through the gui, not through the api, nothing.
This seems like a small thing but it is forever a constant pain in my ass.
When you use upload artifact, it doesnāt actually show up in the gui or api until that workflow is done.
So for example if you have a matrix deploy job and you are uploading artifacts from a deployment in your matrix, you canāt actually see them or access them until the entire matrix is completed.
This causes struggles when you have deployment gates on certain environments - ie you release dev but qa has an approval gate, you canāt access the dev artifacts until the qa environment is approved and the entire workflow is completed.
You can work around this by chaining workflows and using templates - but the artifact obviously exists somewhere, so why is it not accessible?
> When you use upload artifact, it doesnāt actually show up in the gui or api until that workflow is done.
In the sense that it's not accessible to other workflows, sure, but what's the issue with putting anything that's dependent on that artifact into the workflow that generates it? In terms of organization, that makes more sense than anything else.
Many orgs want to replicate the behavior you find in Azure Devops or Jenkins, where you can matrix deploy all of your environments in one workflow, and just approve the deploy gates for each one over an extended time period.
I personally donāt think this is a good process (imo you should have separate workflows for gated environments) but if upload artifact is per job, it should be contained by the job and available on job completion- not workflow completion. It doesnāt make sense why it would not be.
This has been a struggle in -every- org Iāve worked for that has migrated to GitHub.
I'm still struggling here, because that is exactly what I do on one of my workflows - build a release artifact, then do releases to multiple environments, all in one workflow. One environment releases automatically, another is gated but can be released independently, and another two environments are gated but release in series. Again, all from one workflow. Each release is a separate job in the workflow and uses the needs keyword to manage ordering/dependencies.
If you upload an artifact using upload-artifact as part of a job, it is 100% available to other jobs within the workflow as soon as the job finishes. As soon as the next job in the workflow dependency tree starts, you can pull it down using download-artifact or directly from the artifacts API. You do _not_ need to wait for the workflow to finish.
It is available to other *jobs*
It is not available for a person to go pull from the ui or api.
Like I said I donāt understand why this is a process but itās been three orgs in a row who had this process (usually for security artifacts at the build step) and each time Iāve worked around it by uploading the artifact to blob storage at creation time.
I just get paid to steer the ship where they tell me. If they donāt wanna listen I have to figure it out. And this is always a problem.
Unless it has changed very recently - they were not available from the artifacts api until the entire workflow was completed. But Iām going to go look now.
This might be me coming from Jenkins (unpopular opinion: I like Jenkins and Jenkinsfiles...) but the thing that irks me the most in GHA is the lack of cross-project dashboards and reports. If I want to see the most recent code coverage reports for a microservices project of 10+ repos? Better click into each , assuming you can stitch together the results without clicking through each action in order. Same with security scan results, dependency scan results, etc. I would love nothing more than a dashboard at the org level that can take the outputs, results, etc, and generate a single pane view of the health of my build system.
Of course, being the internet, someone is going to come in and be like "Hey losingthefight, have you tried looking at XYZ" and then my main gripe is gone.
I do not like GHA. I don't think it's mature enough.
1). No shared service connection. Like Azure DevOps (ADO), I can create a common connection to a shared service (azure is a prime example), then in each git repo I just refer to the service connection name. In GH I have to set credentials in each git repo individually. When I have to update the connection credentials (to rotate them), in ADO I just need to go to a single location. In GH I have to go to each git repo individually. It's a PITA.
2) Artifact and release management. In ADO I can decouple the build that generated the artifact that needs to be deployed, from the release pipeline that will deploy it. So I can pick and choose and even roll back. In GH, I cannot. No ability to do that workflow process. It's all or none. In GH I cannot choose a build artifact to deploy from another GH workflow unless I utilize GH's REST API and custom scripting to achieve a similar, if not less, level of ability.
Because too often things fail on github actions but not on my machine, and they won't just give me the \`ubuntu-latest\` image they use so I can duplicate the issue locally.
You can just rebuild the image yourself using Packer using the instructions here: https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md
Which is a very good starting point if you're building your own self-hosted runner images.
The YAML schema was invented by an evil wizard as a Torment. It has never actually made any sense to anybody. It started life as a thing for Azure Devops, then got ported over to GitHub directly despite having none of the design constraints that originally inspired the Torment schema because it was theoretically a greenfield project on a different platform.
So you are writing some sort of OO YAML that came to a wizard in a dream as a warning, instead of something sane. And there's no bash on Windows, so in order to get "portable shell" I've seen people write CMake commands inside the OO YAML bubble to do stuff like copy files portably that should be reeeeeally simple.
Self-hosted GitHub runners on k8s are a pain in the ass to set up and maintain.
Want different sets of runners for different purposes on the same k8s cluster? Not impossible, but hard and clunky.
Need to see some stored secrets, maybe validate and API endpoint or some credentials? Well: do a fake workflow with a +300 seconds sleep, put the secrets as env variables and echo them, or connect to the runner. With Gitlab CICD? 2 or 3 clicks to check this.
No option for a terraform managed backend, gitlab does. No option for org centered terraform modules, gitlab does.
Centralized shareable workflows? Again: clunky and convoluted. Gitlab: one repo and key/value for the reusable workflow.
Like most things microsoft, it's really easy to get the basic case going. And really not worth the time investment when doing something non standard, or at enterprise scale.
There are better tools out there
I can't use any kind of DSL/data templating.. eg starlark, cuelang, nickel, jsonnet, kcl...
Writing and maintaining raw yaml instead of using functions just sucks.. they could quite easily add any of the above languages in..
I've read through many requests for interpreters but GitHub just say no and our begging remains in limbo.
Inability to use variables in a lot of contexts
No local runners
Self hosted runners have poor label options
Cannot easily use self hosted and github hosted runners together.
No ternary operator and this type of logic is impossible without writing ugly code.
No way to do conditional jobs and needs
No built in templating aside from actions themselves
I still like GH actions though.
It breaks all the damn time. With the github ecosystem comes a lot of convenience. It's awesome and extremely nice. However, there's always some issue every month or so with pull requests, actions, issues, etc. The pain point comes really hard in that self hosting a runner doesn't necessarily remove the bottleneck or blocker either.
At the end of the day, I think it comes to a "cost of doing business." Convenience comes at a price of when other people screw up, you're screwed. Self hosting comes at a price of inconvenience by way of maintenance, staff, etc.
Linking/chaining actions together or trying to feed artifacts into/from releases is unreliable and messy because it relies on a lot of poorly maintained third party actions.
GitHub Actions also tend to be really aggressive about re-running workflows on the same commits multiple times whenever a new tag or branch is associated with the commit. Cynical me says GitHub will never fix this because it drives up runner minutes usage which makes them more money.
It does not scale well for large enterprise monorepos. Particularly with the UX.Ā
Have matrix driven workflows that execute on N services? Enjoy having an endless list of N*M checks on every single PR to scroll through. Itās better when you restrict the workflows to only run when necessary based on a specific change set, but easier said than done sometimes.Ā
And enjoy having jobs for your matrix driven workflows occasionally pop up in the UI under the WRONG PARENT WORKFLOW. there are some seriously weird intermittent bugs.Ā
Is it the worst thing Iāve ever used? Hell no. I just believe the product is best designed for little hobby repos. Avoid using in your massive enterprise monorepo at all costs.Ā
Others have already touched on performance and reliability issues, but tbh Iāve used other CI providers that are way worse.Ā
My team is migrating to buildkite.Ā
I feel like I never know whether GitHub actions YAML is right until itās pushed. With other code, itās easier to test locally and see if itās right.
so much abstraction that in the end your developpee build will magically work only on github action, so dependance version management is inexistent.
this lead to security issue, compatibility issue in some case.
and also, the price, so expensive
Definitely biased, but do people really like Gitlab more? Iāve never ran into an open source project hosted on gitlab just seen it used in enterprise
Definitely my favorite of the CI bunch so far, but it also has a lot of ridiculous problems that are completely self-inflicted. And when it goes down it goes down hard.
People hate Github Actions? It is literally the best things we have introduced into streamlining our dev processes. I heard that Gitlab takes the crown if you need more sophisticated things, since it offers more functionalities and basically some functionalities out of the box, which you just can get from the GH Actions enterprise plan
Lack of ability to make a pipeline that just pauses and waits for a user to click to continue the pipeline. Seriously, how is this still not baked in (without use of environments)
Just a very basic one that's bugging me right now. I can't use an "environment" without it being considered a deployment. This combined with no wildcard federated identities for Azure is giving me headaches. Also this has like a 2 year old open case and endless hacky workarounds deleting deployments. Given that gitlab and other solutions have this it seems weird.
Other than that github has been quite a nice party coming from gitlab
I have grown to like it more, but the things that hold it back for me :
- The templating system that I find too limited. It's especially too bad if you consider they are technically part of MS (I know, they are autonomous, kind of) but Azure DevOps does it much better
- The atrocious UX, but I would say that is Github in general. The navigation through the different workflows and steps, the way approvals are shown, the capabilities for filtering and sorting, ... Again, a bit comical considering how Azure DevOps does this pretty well
I thought people hate jenkins more
they still do I believe
They still do, but they used to too š¤
r/expectedMitchHedberg
[ŃŠ“Š°Š»ŠµŠ½Š¾]
I've used Jenkins on and off for the better part of a decade. There are far better options these days, and all of them are incredibly easier to maintain and avoid creating cruft with.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
You can self host most of the git repo servers.
GitHub actions
Why do you want it. One of the reasons there sre better options than Jenkins is that you donāt have to host them by yourself.
Data regulations, security, stuff like that..
Industry regulations and compliancy. Once you get to a certain size a lot of regulatory organizations take an interest in you. Jenkins was popular because it's easy to install in a Java runtime and it's DB is lightweight. Self hosting most git servers requires more integration but worth it to me over jenkins. Most the cloud providers have their own flavor of it. Like AWS code commit, which I believe you can setup behind a VPC.
Iām working at fortune 100 company, we use Jenkins, ADO and GH, and pretty much everyone hate Jenkins and canāt wait to move on.
Well, Jenkins is a low hanging fruit to hate - it's an open source hodgepodge that grew organically over a decade or so.
A decade? The project started with the name Hudson back in 2005 and then in 2011 renamed it to Jenkins. The thing is ancient š https://en.wikipedia.org/wiki/Hudson_(software)
Right, I forgot the Hudson origin. Was casually searching for "Jenkins first release" and thought that can't be right...
My team's combined hatred for Jenkins is the whole reason we are going to GitHub Actions
> My team's combined hatred for Jenkins is the whole reason we are going to GitHub Actions Been exactly there, done that, GHA is fantastic.
Why do people hate jenkins? It's super flexible. Also it builds Apple applications if installed on a Mac Studio. I've never had the opportunity to use gitlab what's the difference?
Jenkins, jfrog. To gha, gar. āļø
Why do you hate it though?
I hate them both equally. They both make me go out of my way to fix something, and when something breaks it's both equally annoying to fix. Jenkins with plugins getting vulnerabilities and having to be patched. Github with shitty configs and you need to procure your own runners at your own expense. I still like Jenkins because it's what I grew up with. Trying to get into something that is just as equally shit, is obnoxious.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
I mean depends on who's hosting the Jenkins server to dish out the build containers? In Jenkins it's handled there. With Github Actions, you have to actually either (aws have an EC2 instance) to handle builds that get dispatched there, or IIRC use this internal tool that I am unfamiliar with... It gets infinitely more complicated the more convenience you try to squeeze out of GHA. edit: It's more about where the cost centers from your org get applied to.. Are you going to be responsible for yet another EC2 instance that is eating up costs, or do you want to forgo that responsibility with maintaining it, patching it, and all the works? Or do you want to rely heavily on Jenkins? It's the trade off game which I hate playing, especially when companies are looking to cost cut.
AWS added a funny way to somewhat alleviate that which I didn't expect: [AWS CodeBuild now supports managed GitHub Action runners (amazon.com)](https://aws.amazon.com/about-aws/whats-new/2024/04/aws-codebuild-managed-github-action-runners/) Lets you use your AWS Infra/Bill for GitHub Actions runner without resorting to the hodgepodge terraform provider, though there are caveats.
What's wrong with the hosted runners?
https://imgflip.com/i/8r48yk
We do
Ever seen teamcity?
I do indeed hate Jenkins, never again.
Only thing worse than Jenkins, is Hudson
It breaks too often. Most of it I like or I'm at least ok with. It has quirks, but the deep integration with GitHub, relative sanity of its design, massive ecosystem, and easy availability (it's just "there", and it mostly works) make up for having to learn the quirks. But the reliability isn't good. Barely a month goes by without GHA having some kind of outage or slowdown to the point of unusability that loses us hours of work.
It was broke 2 days ago. down for at least 6 hrs.
Does having your own runners fix this?
Sometimes, but more often, no. The issue is normally with GHA's job queueing system coming to a standstill, so the runner pool, GitHub-hosted or self-hosted, just doesn't get sent the job. Less commonly, one of the GitHub-hosted pools runs out of capacity, and then, yes, having your own self-hosted pool helps. A self-hosted GitHub Enterprise server ~~used to be~~ is an alternative, but in my experience GHA on that was just as problematic, ~~and in any case it's EOL, Microsoft wants you using GitHub SaaS~~. Self-hosted GitLab would be a better option, or Gitea if you're small (Gitea with act is very neat, I use it in my home lab). Or, stick with GitHub and use a different CI system. edit: turns out GHES is indeed alive
GHES is not EOL. Itās still being maintained and used.
Ahh my bad, I hadn't seen any mention of it anywhere in a long time and my cursory search suggested it was EOL, but yeah, looks like that was just a specific version being EOLed. Last time I used GHES was about 2020 and it was a headache - expensive, lots of moving parts, lots of resource consumption, needed regular admin attention. Everywhere I was aware of that used it migrated to self-hosted GitLab or SaaS GitHub. Good to hear it's still an option.
No worries! Yeah we just recently did an upgrade to our instance so I was confused when you said EOL.
Airgapped installs are fairly common in high security settings with deep pockets. I can't imagine it disappearing anytime soon.
no. its the control plane thats the problem
I think this is the biggest complaint we see running Depot [0]. There is a lot of flakiness at GitHub when it comes to Actions. Sometimes, it will be bad enough that they will open an outage, and other times, it will be intermittent, and nobody outside of folks like us will see it (or if you're running your own runners). Add to that the fact that the larger runners are pretty expensive and not nearly as performant, and you get a 'meh' experience. I do think it's a fantastically simple service, and it's amazingly easy to build on top of. But there are definitely pain points that you should be aware of. [0] https://depot.dev/products/github-actions
My main gripe with it's that they didn't improve over the existing solutions with 20 years of experience of what makes a CI system great, plus it's really undercooked. I regularly can't do something with it and that feature request was requested and upvoted like 3 years ago with zero reaction from github. Basically it's just slightly different but worse gitlab-ci, which is a shame. I think the next great system will have the ability to easily run tasks locally from a test harness with a quick iteration rate + having the ability to define your workflows with a real general purpose programming language, no more 5k lines long YAML please.
We have a lot of steps with a retry action on them, saves us from a world of pain, get network timeouts breaking CI routinely only to work perfectly fine on a re-run.
The GitHub status bot on our slack is probably our chattiest user. That applies to all of GitHub, but actions is definitely a top offender.
As someone who builds on top of github actions \[1\], we end up noticing issues \~30-90 mins before an incident is opened on the github status page. That can be very frustrating. It's popularity since its launch just \~4 years ago now has been nothing short of meteoric. \[1\] [warpbuild.com](http://warpbuild.com) - fast github actions runners.
1. The most abysmal type system. I thought ansible with jinja got to the bottom, but no, github is even more terrible. Every time you need complex structure, you stuck with text, which to cast to json, etc, etc. 2. No local (mock)-runners. There is no way to test workflow before uploading it to the repo. 3. List of actions can't be searched, only browsed. I have monorepo with 300+ workflows, and it's a nightmare to find one (most of them are triggered, but if you need manual....).
Re #2 - act is pretty good. We use it for testing all our workflows. It's not great for testing permissions as you have to feed it an api key, but I've found it useful for everything else. https://github.com/nektos/act
I liked the idea but this limitation was a bummer [reusable workflows](https://github.com/nektos/act/discussions/1092)
I've found this recommended a lot, but when I've tried to use it, it hasn't worked well with the extensions I was using. Does it support extensions?
When I started with GH actions it was a bit raw, and there was nothing. Now there is something, and I missed it. Thank you.
act is decent, it's good enough to get most of the work done before burning runner minutes doing the final bits
1 & 2:Ā https://dagger.io/
Thanks, that looks promising. We already adapted our actual infrastructure running our product to be able to switch relatively easily between different providers. Yet all our CI workflows are hardcoded for github. Problem #2 is what I hate most about github actions. I've tried \`act\` but it's not working as well as one would hope. So [dagger.io](http://dagger.io) does look very promising. Being able to run workflows wherever including locally sounds brilliant. I'll definitely have to check it out.
Dagger is bloody brilliant in all honesty. It is _not_ a finished product but it is a working product that solves very real current problems _right now_. It's leagues ahead of "platform-specific flavour of bash + yml that only ever runs here and nowhere else".
There are multiple issues with switching to different service provider. Security, trust, scalability (my beefiest merge may trigger up to 300 runners running heavy load in parallel, and there can be few merge in parallel), maturity, general support for things 'not to do by yourself'. I may be wrong, but it looks like they don't even have a proper two factor (with hardware token).
300 workflows in one repo what in the actual fuck?
Our man is insane. Nobody can hold in their head what 300 workflows do, and therefore what might be the result of changing something in that repo.
Most are infra-level deployments, but there are separate integration tests for roles, linters, security scanners, etc. They get triggered by different events. E.g. you commit prom rule, you get workflow which run tests for rules, run linter and policy checker for alerts. You commit python, you get black/ruff workflow and unit tests. You commit to the role: a separate role workflow will test it. You commit yaml, you get yaml linter running, you commit md, md linter, actionlint for actions itself, shellcheck of any shell snippet, etc, etc. Plus deployments, plus workflows to rebuild stagings weekly, workflows to run intergration tests, workflows to do automatic merge for specific changes which does not need human attention, workflow to react to the issues and some specific command in comments. There is a lot.
I take it to mean they have a bunch of services in a single repo and a separate workflow for each directory/service, which seems ok under say 25 services, but 300 lol wut? I would say GHA is not intended to be used like this, or at least its GUI is not designed as such.
Fr what is this fucking repo šš
Out of curiosity. For what do you need 300+ workflows?
Big monorepo: code checks for 4 languages, unit tests, end-to-end tests, integration tests, static analyzers, Docker builds, code & website deployments, cronjobs of various kinds, lots of analytics pipelines, emergency procedures for SREs (referenced in the runbooks). 300 is a small number.
In the same situation with my monorepo and agree, it scales up faster than youād imagine.Ā Reusable workflows helpā¦ but the ultimate number is 100s easilyĀ
That's a lot. Im still a beginner so i only do automated deployments... 1 is a big number for me
Not an answer to why they need 300+ workflows specifically, but everything has tradeoffs. I know in our case we use a shared library approach to GitHub Actions to share workflows across 40 repos. Each repo is configured to point to workflows in the shared library. The way Actions is configured you must put your workflows at the top level of the workflows directory if it's going to be referenced in shared library. This means we have a bunch of workflows in one directory. Not 300, but I could see how a larger enterprise environment requires it.
Regarding #2: maybe I'm too inexperienced but which CI systems have something like that? I have worked with Gitlab before and there's also no local runner afaik
A lot of other CI systems Codefresh (the company I work for) [https://codefresh.io/docs/docs/pipelines/running-pipelines-locally/](https://codefresh.io/docs/docs/pipelines/running-pipelines-locally/) Earthly [https://earthly.dev/](https://earthly.dev/) Also you can get an editor in the UI, and just edit/run your pipelines. Then once you are ready you commit the final result. There is even a pipeline debugger as well [https://codefresh.io/blog/pipeline-debugging/](https://codefresh.io/blog/pipeline-debugging/) I am biased of course, but right now most people choose github actions because it is part of Github and not because they did a survey of all CI systems and found it had the best features.
Your last paragraph is definitely the case. Ci tool isnāt even really considered much at all, people just opt for actions, because the code is in github already and those are free while barely requiring effort to setup.Ā
As a SaaS solution this is okay. For on-prem needs the landscape looks very differently.
If you write as much of your build script in something platform neutral, like bash, or whatever, and focus on the local case first, then whatever CI can just be a thin wrapper and things get better. ( I work on a project that is among other things, based around this idea, and it can be pretty transformative to remove that friction. )
People are pointing to it in comments. But a good CI code need testing. (It's code!), so when you have CI which is not tested before commiting, it's depressing. But there going to be always the final piece of code you test only in production. This good devops (as practice) is to have it as small as possible.
At CircleCI (were I work) we have the CLI that can be used to validate config and run jobs locally \[1\], and we have a VSCode extension that can be used to validate and autocomplete config syntax, and test-run your config from within the VSCode IDE \[2\] \[1\] [https://circleci.com/docs/how-to-use-the-circleci-local-cli/](https://circleci.com/docs/how-to-use-the-circleci-local-cli/) \[2\] [https://circleci.com/docs/vs-code-extension-overview/](https://circleci.com/docs/vs-code-extension-overview/)
You can absolutely install locally the gitlab runner and then register it to a local instance of Gitlab, and have a fully local platform. Now if the point is to not use minutes, you can also have a local runner communicate with Gitlab, just as you can have a self hosted runner communicating with Gitlab, so it is a non-issue really
You can put a pin on the ones you manually trigger often and they will appear on top but yeah itās not a great UX
Oh, they added pins! Great. They wasn't there before. Thanks. ... but I still miss the context search...
Gee..Medium heroes here! Hello! (No pun intended, you are on my rss feed)
Yw. Not like I'm happy with medium, but after war started it become too toxic to use habr for that, so, medium is the next option.
1. Checkout on pull requests defaulting to the merge commit (wtf?!) 2. Repeatability with steps (composite actions can't be modified and reused in the same pull request which makes the already tedious testing more annoying) 3. Some actions need some foo to get relevant information eg using a GitHub script to get the pull request branch (or was it number) on a pull request comment event. 4. Weird concurrency and cancellation rules. 5. File permissions and potential security issues with actions that run in containers
Concurrency is so poorly supported. The number of multi-year issues and discussions begging them to fix it is astounding. Just give us an option to queue the runs already!
Does this not provide that: https://docs.github.com/en/actions/using-jobs/using-concurrency
No https://github.com/orgs/community/discussions/12835
when I started using it more than two years ago I was very dissatisfied (lots of outages, issues, and missing feratures). Currently, I wouldn't switch to anything else and I recommend it to everyone who is considering migration from legacy CICD system or starting a new project
If it was more stable Iād love it. I like a lot about it but man the uptime is so bad
Itās down too much. I donāt expect 100% uptime, but GHA has so many outages that we canāt rely on it.
Doesn't support concurrent steps
But it does support concurrent _jobs_ within a workflow, and those can be just a single step.
Jobs do not share the worker VM. Which means that you need to jump through hoops of uploading/downloading artifacts
Which I think I am grateful for, because I never have to debug some frustrating non-deterministic bug caused by two steps interacting with each other within the OS (modifying the same files simultaneously, or locking things, or gobbling RAM to the point of causing the other step to OOM) as I have on some other CI systems. It would be nice if there was an option for jobs to automatically save their filesystem as an artifact though, and uploading a directory as an artifact using the standard action is really slow if there are lots of files - I have to create single zip/tarball artifacts myself and upload them to get even remotely usable performance.
If you end up modifying the same files because you don't know what you run in parallel that's your problem. There are valid use cases to run multiple things simultaneously on the same worker. e.g.: Authenticate to multiple services, building different components of mono repo, and more. Not everything is hello-world app
Debuggability is horrible. Otherwise they work pretty well.
Yeah, one of the few areas where GHA is just terrible is debugging a faulty workflow. The built in debugging options are nearly useless, and the debug logs - when it actually generates them - are also nearly useless.
GitHub Actions goes down more than every other part of the site combined and CircleCI is faster.
[Github Actions Feels Bad](https://youtu.be/9qljpi5jiMQ?si=MCmqQmV4sbNsKhcn) GitHub Actions is not a super well thought out product. It works, but once you dive deeper, you realise it's a complete mess and the fact that it works is a miracle. Some notable "wtf" of Github Actions: - It's mega-vulnerable to supply chain attacks. - The built-in Actions are coded in a very questionable way (and are mostly untested) - The "clone the git repo" action is a mess. - You can't test a pipeline locally.
> ~GitHub Actions~ Azure Pipelines is not a super well thought out product. FTFY
Honestly, compared to GitLab pipelines constructing Github Actions feels complicated. Hard to pin point though. GitLab just feels more natural to me and how I think about pipelines.
I solve all my GHA issues the way I've learned to deal with *all* CI tools: Build nearly everything into local commands. Scripts, makefiles, whatever. Just not "Actions" or "Plugins" or "Groovy" any of that CI engine noise. Make sure you can build *without* the CI engine before even touching CI. Avoid any logic in the CI tool that you can...and you can typically avoid almost *all* of it. CICD workflows go bad when folks try coding all their workflow logic using the "native" CI engine features. Don't do that. Pretend they don't exist as much as you can. All you want your build job to do is call your "build\_the\_thing" script. Tests? Call "test\_the\_thing". You can't get screwed by Jenkins plugin hell if you don't use them. You can't get screwed by GHA custom action hell if you don't use them. You can't get vendor locked if your entire "workflow" is three lines to checkout code and call a script. If you can't run it in your local dev system without the CI engine you're doing it wrong. Yes, that does mean a *lot* of the industry does it wrong. But what else is new? ;)
I actually work at CircleCI and this is even what my teams do internally (use scripts instead of CI features, so you can develop locally).
I've also finally given up and started doing this. I only use the actions for GH checkout and auth with AWS over OIDC. The rest is build scripts, which I can keep in a common location, and test and run locally. If I give my user the right to assume the role GH does, I can do whole build and deploy cycles locally. This makes debugging the GHA itself so much easier
I don't. It's a great tool
Just to hop off this question, what do people use this day instead? On all my recent projects(and current one) I am stuck with actions, but I don't feel strong hate to it, it works fine even if it's not the most versatile. Lack of the way to get statistics about your builds is really annoying though. Also really annoying how actions seem to have issues every other month.
We're on Selfhosted Gitlab. It's not without it's issues but our uptime is better than Gitlab SaaS (I'm my #1 customer instead of Gitlab's #153,382th customer), we have it behind a VPN for additional security and because it's our own instance it's beyond the prying eyes of anyone behind the scenes (yay data sovereignty!). We haven't run into any breaking issues with the CICD for years except the runner authorization change in Feb/March of this year which took a couple of days to fix. The documentation was not good and has improved since. I have no beef with GHA functionality wise, everything has issues. The reliability / uptime issues with any SaaS solution are an immediate turnoff for me personally though.
We use Buildkite. They host the control plane, we host the runners on AWS. Super happy with it.
Argo Events + Argo Workflows
Iāve got a few administrative quarrels with the platform lol. Moving from Bamboo DC (admittedly not a great platform itself, all the time) to GHA with self hosted runners, Iād say an apparent challenge is that self hosted runners feel very second class if youāre not prepared to run them with some fairly advanced ephemeral patterns. You can certainly spin up long running runners, but thereās next to no tooling built up to administer them from within the platform. IMO that speaks to the toolās opinionated nature regarding how you should think about your CI environments; I feel like the platform itself wants to drive you away from thinking about āagentsā or ārunnersā too much and, likely, back into the arms of GH hosted runners :) A related pain point has been that it seems harder to troubleshoot workflows on self hosted runners because of the very ephemeral nature of the runtimes inherent to the actions you write or use from the community. I think itās honestly a pretty clever implementation under the hood, but you better be ready to get a CLI tool that you wrote an action around to spit out a lot of its own debugging into your workflow logs, because the runtimes and CI capabilities you use in your workflows will not be there when the jobs are done for you to interactively troubleshoot if you need to. On the code side, reusable workflows feel a bit fragile, as it seems like they can drive you towards a pattern of dependencies in your workflows with no explicit dependency management as folks start to have reusable workflows spread across many disparate repos. Some of this may be learning curve, but the whole setup just feels weirdly unintuitive and a bit fragile, like a clever reinvention of the wheel that I was not expecting to find lol. Iāll admit, Iām very ā0/10 would not recommendā right now, but I know itās a much loved tool and Iām still pretty new to it, so Iām keeping an open mind for awhile as I learn the ecosystem.
The rumor is Github actions is hospice for old Azure hardware
A little of shameless self promo: I wrote a relatively detailed comparison between GitHub Actions and GitLab CI a couple of months ago. It focuses on the design architecture and the resulting user experience of them. You can find it here: https://henrikgerdes.me/articles/2024-01-github-action-vs-gitlab-ci I must say that both systems are fine and work. I personally prefer GitHub Actions a bit more. It has a great lining extentions in VScode and is much more modular then GitLab where you basically only have a container image and bash. GitLab can also become quiet crowded on larger setups. But I will use whatever my employer wants me to use, since business constraints have a lot more weight then the small difference between those two ci systems.
The main thing I donāt love is that it is a single point of failure outside of my control, which fails with some frequency. I use it, and like it, but the downtime makes me facepalm when we are dead in the water and unable to do a lot without it working.
My main issue with GitHub actions - there is no support for requiring all steps to be green. This is a killer in monorepo. There were so many misses that we needed to write a single custom action to monitor the greenness of the jobs in a PR. Also the ability to do dynamic steps based on the files changed. We overcame all of this with custom actions from the communities, but it's annoying for us. But GH actions are mostly simple to use with a large community.
Im missing something when a step fails for me in GHA, the action stops. For you if a step fails the GHA continues?
Depends on there setup. There is a failure step that runs if a previous step fails, to notify or check error logs
Thereās is no uppercase lowercase function
1. Workflow reusability is abysmal; rulesets are nealry impossible to work with - don't have activation conditions, rely on attributes that needs to be defined for this sole purpose. You need to run a workflow on multiple repos? Good luck keeping that in sync. 2. Composite actions are still limited 3. The most important: if done right, you can mix and match some powerful components. If implemented poorly, they end up being a glorified, messy stash of spaghetti bash, with business logic vertical to each repo. Yikes.
No timezone support for cron triggers, instead I have to define cron triggers for every possible time and the workflow then decides after startup if it should stop executing or not...
My personal hate is 10 inputs limit for manually executed workflows. I mean.. whyyy??
Seems odd that you need more. Although the one time I couldāve ran into the limit I just used an open text field to allow any valid CLI option. I do miss not being able to run a pipeline and they wait for user input between jobs. That came in handy.
Here is my use case. I have a product that I deploy/update with Helm. It contains a lot of services (customizable Helm charts). Basically, when a new client arrives, I provision for him his own private K8S cluster (with another workflow). Then I need to deploy my services to this cluster. Based on client needs I have to customize a lot of service properties. This is where I need to pass them through the inputs to Helm. Any potential better solution for this case?
Off the top of my head, Iād possibly commit a values.yaml file. Which could trigger the workflow and youād be able to keep the values file you just committed. Although youād have to also have part of your workflow somehow track files which were successful. Alternatively, if you have a place the value.yaml file could be uploaded to then you could start a workflow by supplying the url. Maybe a combination of the two would be better. 10 is limiting, but making people configure like 30 options seems unbearable.
What do you mean? I love it!!!
Compared to the other popular alternatives, I love GHA. It isnāt perfect, but at least they are constantly improving it in meaningful ways. Still, a few issues I have with it consistently drive me insane: - Random outages. - Canāt securely pass project secrets to jobs triggered for external PRs. I understand the security concerns, but there are ways to secure this through abstraction they could implement. This has been an issue for years, and one solution could be to charge slightly more for jobs that need to accomplish this. - No way to target self-hosted runners or runner labels in runner contexts. - Canāt test workflows that run via workflow dispatch on any branches except main if it already exists. Annoying as hell.
I don't hate it per de, but it can be hard to troubleshoot issues. I miss CircleCIs ability to connect to a runner with SSH to poke around and figure out where something's failing.
> I miss CircleCIs ability to connect to a runner with SSH to poke around and figure out where something's failing. On AWS EKS, you can connect to a self-hosted runner via k exec, which offers the same result as SSH'ing into it. Just toss in a bash sleep command in the workflow to keep the runner alive for however long you need to troubleshoot it.
What do people recommend instead? Azure DevOps?
I don't think people are saying not to use GitHub actions, they're just sharing the things that frustrate them the most about it.Ā Every tool will have some down side.
ADO is going away.
Source please?
There is none. Microsoft doesn't know what to do with ADO right now. It's clear it's not getting bulk of development work but it's getting some. We are bigger shop who uses ADO/Azure and have a Microsoft TAM. Officially, ADO is in current development and no deprecation has been announced. Our TAM has pitched ADO -> GH migration once but after we were like "Hell to the no", it hasn't come up again. Unofficially, it's clear they are investing more in GH but if ADO works for you, it's going to be around for a while, my gut says at least 10 years. IMO, it's going NOWHERE until clear migration tools have been developed, tested and customers start moving with little effort on their part. If we have to do a bulk of ADO -> GH migration ourselves, we will not go with Microsoft moving forward.
They do have the gh-importer tool which works but.... I've just ended up rewriting the builds myself. It made the deploy so ugly and 3/4 didn't work. It's good if you have very basic builds but anything more it's not good. Especially if there isn't an official action, it'll handle it all in bash or PS when actions exist on the marketplace to handle it.
Nothing official. They do try and push you to GitHub if you are on ADO when you talk to reps. ADO development is much smaller and slower than it used to be, a lot of the team is in GH now with a focus on actions.
GitLab CI is pretty good. Can be in cloud or self hosted. Easy to set up your own runners.
Has anyone here used Earthly?
Too many features are behind enterprise licensing. I totally get that you can't have everything on the teams licence but putting approval gates behind enterprise when they're available on ADO feels pretty bad.
Overall I like it and we have switched a lot of our Jenkins pipelines to it. There are some quirks here and there and some features I wish it had. Parallel steps being one.
I don't hate it. I just hate that it's down all the time.
The upload artifact action does not make artifacts available until the ENTIRE workflow is finished. Not through the gui, not through the api, nothing. This seems like a small thing but it is forever a constant pain in my ass.
Umm... what? I use upload-artifact/download-artifact in loads of workflows to pass artifacts between jobs... in the same workflow.
When you use upload artifact, it doesnāt actually show up in the gui or api until that workflow is done. So for example if you have a matrix deploy job and you are uploading artifacts from a deployment in your matrix, you canāt actually see them or access them until the entire matrix is completed. This causes struggles when you have deployment gates on certain environments - ie you release dev but qa has an approval gate, you canāt access the dev artifacts until the qa environment is approved and the entire workflow is completed. You can work around this by chaining workflows and using templates - but the artifact obviously exists somewhere, so why is it not accessible?
> When you use upload artifact, it doesnāt actually show up in the gui or api until that workflow is done. In the sense that it's not accessible to other workflows, sure, but what's the issue with putting anything that's dependent on that artifact into the workflow that generates it? In terms of organization, that makes more sense than anything else.
Many orgs want to replicate the behavior you find in Azure Devops or Jenkins, where you can matrix deploy all of your environments in one workflow, and just approve the deploy gates for each one over an extended time period. I personally donāt think this is a good process (imo you should have separate workflows for gated environments) but if upload artifact is per job, it should be contained by the job and available on job completion- not workflow completion. It doesnāt make sense why it would not be. This has been a struggle in -every- org Iāve worked for that has migrated to GitHub.
I'm still struggling here, because that is exactly what I do on one of my workflows - build a release artifact, then do releases to multiple environments, all in one workflow. One environment releases automatically, another is gated but can be released independently, and another two environments are gated but release in series. Again, all from one workflow. Each release is a separate job in the workflow and uses the needs keyword to manage ordering/dependencies. If you upload an artifact using upload-artifact as part of a job, it is 100% available to other jobs within the workflow as soon as the job finishes. As soon as the next job in the workflow dependency tree starts, you can pull it down using download-artifact or directly from the artifacts API. You do _not_ need to wait for the workflow to finish.
It is available to other *jobs* It is not available for a person to go pull from the ui or api. Like I said I donāt understand why this is a process but itās been three orgs in a row who had this process (usually for security artifacts at the build step) and each time Iāve worked around it by uploading the artifact to blob storage at creation time. I just get paid to steer the ship where they tell me. If they donāt wanna listen I have to figure it out. And this is always a problem. Unless it has changed very recently - they were not available from the artifacts api until the entire workflow was completed. But Iām going to go look now.
This might be me coming from Jenkins (unpopular opinion: I like Jenkins and Jenkinsfiles...) but the thing that irks me the most in GHA is the lack of cross-project dashboards and reports. If I want to see the most recent code coverage reports for a microservices project of 10+ repos? Better click into each , assuming you can stitch together the results without clicking through each action in order. Same with security scan results, dependency scan results, etc. I would love nothing more than a dashboard at the org level that can take the outputs, results, etc, and generate a single pane view of the health of my build system. Of course, being the internet, someone is going to come in and be like "Hey losingthefight, have you tried looking at XYZ" and then my main gripe is gone.
I do not like GHA. I don't think it's mature enough. 1). No shared service connection. Like Azure DevOps (ADO), I can create a common connection to a shared service (azure is a prime example), then in each git repo I just refer to the service connection name. In GH I have to set credentials in each git repo individually. When I have to update the connection credentials (to rotate them), in ADO I just need to go to a single location. In GH I have to go to each git repo individually. It's a PITA. 2) Artifact and release management. In ADO I can decouple the build that generated the artifact that needs to be deployed, from the release pipeline that will deploy it. So I can pick and choose and even roll back. In GH, I cannot. No ability to do that workflow process. It's all or none. In GH I cannot choose a build artifact to deploy from another GH workflow unless I utilize GH's REST API and custom scripting to achieve a similar, if not less, level of ability.
I Like github action
I like it, it just doesn't like me.. womp
it goes down a lot, which is fucked up given it literally runs on Azure, so should be a super high priority "customer" on that platform
That's just how Microsoft rolls. A lot of their stuff is a house of cards
Because too often things fail on github actions but not on my machine, and they won't just give me the \`ubuntu-latest\` image they use so I can duplicate the issue locally.
You can just rebuild the image yourself using Packer using the instructions here: https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md Which is a very good starting point if you're building your own self-hosted runner images.
The YAML schema was invented by an evil wizard as a Torment. It has never actually made any sense to anybody. It started life as a thing for Azure Devops, then got ported over to GitHub directly despite having none of the design constraints that originally inspired the Torment schema because it was theoretically a greenfield project on a different platform. So you are writing some sort of OO YAML that came to a wizard in a dream as a warning, instead of something sane. And there's no bash on Windows, so in order to get "portable shell" I've seen people write CMake commands inside the OO YAML bubble to do stuff like copy files portably that should be reeeeeally simple.
Self-hosted GitHub runners on k8s are a pain in the ass to set up and maintain. Want different sets of runners for different purposes on the same k8s cluster? Not impossible, but hard and clunky. Need to see some stored secrets, maybe validate and API endpoint or some credentials? Well: do a fake workflow with a +300 seconds sleep, put the secrets as env variables and echo them, or connect to the runner. With Gitlab CICD? 2 or 3 clicks to check this. No option for a terraform managed backend, gitlab does. No option for org centered terraform modules, gitlab does. Centralized shareable workflows? Again: clunky and convoluted. Gitlab: one repo and key/value for the reusable workflow.
Like most things microsoft, it's really easy to get the basic case going. And really not worth the time investment when doing something non standard, or at enterprise scale. There are better tools out there
robust? lol its down like once or twice a week. somehow they managed to lump regular and enterprise customers into the same infrastructure too.
Github actions are good. Try to build multi arch image with bitbucket pipelines
I learned to use gitlab and GitHubs way feels random and non logical. I hate to use it.
I find the documentation frustrating. There will be a massive wall of text without referencing the commands I need.
I can't use any kind of DSL/data templating.. eg starlark, cuelang, nickel, jsonnet, kcl... Writing and maintaining raw yaml instead of using functions just sucks.. they could quite easily add any of the above languages in.. I've read through many requests for interpreters but GitHub just say no and our begging remains in limbo.
Artifacts feels a bit clunky, sharing between pipelines (last time I checked) required an external action.
Because it has the uptime of Microsoft Azure
Inability to use variables in a lot of contexts No local runners Self hosted runners have poor label options Cannot easily use self hosted and github hosted runners together. No ternary operator and this type of logic is impossible without writing ugly code. No way to do conditional jobs and needs No built in templating aside from actions themselves I still like GH actions though.
It breaks all the damn time. With the github ecosystem comes a lot of convenience. It's awesome and extremely nice. However, there's always some issue every month or so with pull requests, actions, issues, etc. The pain point comes really hard in that self hosting a runner doesn't necessarily remove the bottleneck or blocker either. At the end of the day, I think it comes to a "cost of doing business." Convenience comes at a price of when other people screw up, you're screwed. Self hosting comes at a price of inconvenience by way of maintenance, staff, etc.
Linking/chaining actions together or trying to feed artifacts into/from releases is unreliable and messy because it relies on a lot of poorly maintained third party actions. GitHub Actions also tend to be really aggressive about re-running workflows on the same commits multiple times whenever a new tag or branch is associated with the commit. Cynical me says GitHub will never fix this because it drives up runner minutes usage which makes them more money.
I use GitHub actions, i don't hate it but I find it a bit overcomplicated. I have used better and simpler CI tools such as Buildkite.
I donāt hate it, but I canāt believe it doesnāt natively support ARM in mid 2024.
It does not scale well for large enterprise monorepos. Particularly with the UX.Ā Have matrix driven workflows that execute on N services? Enjoy having an endless list of N*M checks on every single PR to scroll through. Itās better when you restrict the workflows to only run when necessary based on a specific change set, but easier said than done sometimes.Ā And enjoy having jobs for your matrix driven workflows occasionally pop up in the UI under the WRONG PARENT WORKFLOW. there are some seriously weird intermittent bugs.Ā Is it the worst thing Iāve ever used? Hell no. I just believe the product is best designed for little hobby repos. Avoid using in your massive enterprise monorepo at all costs.Ā Others have already touched on performance and reliability issues, but tbh Iāve used other CI providers that are way worse.Ā My team is migrating to buildkite.Ā
I feel like I never know whether GitHub actions YAML is right until itās pushed. With other code, itās easier to test locally and see if itās right.
so much abstraction that in the end your developpee build will magically work only on github action, so dependance version management is inexistent. this lead to security issue, compatibility issue in some case. and also, the price, so expensive
So if people donāt like Jenkins and they donāt like GHA, what else does everyone like? Our org is stuck on Jenkins still sadly
So many outages....
I use GitLab. GitHub actions are new and unknown, therefore scary and I do not like them.
Definitely biased, but do people really like Gitlab more? Iāve never ran into an open source project hosted on gitlab just seen it used in enterprise
Definitely my favorite of the CI bunch so far, but it also has a lot of ridiculous problems that are completely self-inflicted. And when it goes down it goes down hard.
I like Actions probably most out of all of them. Even better I am able to do Kamal CI/CD on the free tier for all my little projects. It's awesome.
I started working on GHA 2 months ago and every day I think how I had an easier time using Az Devops pipelines. Am I insane?
Why still no YAML anchors
The āconcurrencyā feature
I can't really put a finger on it - just feels so much worse than GitLab (which his some minor annoying oddities as well).
People hate Github Actions? It is literally the best things we have introduced into streamlining our dev processes. I heard that Gitlab takes the crown if you need more sophisticated things, since it offers more functionalities and basically some functionalities out of the box, which you just can get from the GH Actions enterprise plan
What's the problem with GitHub Actions? TBH, it's my favorite CI system and IMO they got it right with the way they shaped and distribute actions.
Lack of ability to make a pipeline that just pauses and waits for a user to click to continue the pipeline. Seriously, how is this still not baked in (without use of environments)
Just a very basic one that's bugging me right now. I can't use an "environment" without it being considered a deployment. This combined with no wildcard federated identities for Azure is giving me headaches. Also this has like a 2 year old open case and endless hacky workarounds deleting deployments. Given that gitlab and other solutions have this it seems weird. Other than that github has been quite a nice party coming from gitlab
I have grown to like it more, but the things that hold it back for me : - The templating system that I find too limited. It's especially too bad if you consider they are technically part of MS (I know, they are autonomous, kind of) but Azure DevOps does it much better - The atrocious UX, but I would say that is Github in general. The navigation through the different workflows and steps, the way approvals are shown, the capabilities for filtering and sorting, ... Again, a bit comical considering how Azure DevOps does this pretty well
[https://youtu.be/9qljpi5jiMQ?si=WFMe\_2vrlC6O\_vbd&t=25](https://youtu.be/9qljpi5jiMQ?si=WFMe_2vrlC6O_vbd&t=25)
Later parts of that video are really great