How to Update your Android Kernel to Latest Linux Stable

We’ve covered guides on Android kernels, such as “How to Build a Custom Kernel” and “Best Custom Kernels for Android”, but today we’re going to show you how to upstream your kernel against the latest Linux stable.

Please know that this is advanced stuff – if you’ve never compiled a kernel before, you should follow the guide “How to Build a Custom Kernel” linked above, and this guide will involve cherry-picking and merging commits from the latest Linux-stable kernel with your Android kernel before you compile it.

Upstreaming your Android kernel to the latest Linux stable has a lot of positive benefits, such as being up-to-date with the latest security commits and bugfixes – we’ll explain some of the pros and cons later in this guide.

What is Linux-Stable kernel?

Linux-stable as the name implies is the stable arm of the Linux kernel. The other arm is known as “mainline”, which is the master branch. All of the Linux kernel development happens in the mainline, and generally follows this process:

1. Linus Torvalds will take a bunch of patches from his maintainers for two weeks.
2. After this two weeks, he releases an rc1 (e.g. 4.14-rc1) kernel.
3. For each week for the next 6-8 weeks, he will release another RC (e.g. 4.14-rc2, 4.14-rc3, etc) kernel, which contains ONLY bug and regression fixes.
4. Once it is deemed stable, it will be released as a tarball for download on org(e.g. 4.14).

What are LTS kernels?

Every year, Greg will pick one kernel and maintain it for either two years (LTS) or six years (extended LTS). These are designed to have products that need stability (like Android phones or other IOT devices). The process is the exact same as above, it just happens for a longer time. There are currently six LTS kernels (which can always be viewed on the kernel.org releases page):

What are the benefits of upstreaming my Android kernel to Linux Stable?

When important vulnerabilities are disclosed/fixed, the stable kernels are the first ones to get them. Thus, your Android kernel will be a lot safer against attacks, security flaws, and just bugs in general.

The Linux stable includes fixes for a lot of drivers my Android device doesn’t use, isn’t this mostly unnecessary?

Yes and no, depending on how you define “mostly”. The Linux kernel may include a lot of code that goes unused in the Android system, but that doesn’t guarantee there will be no conflicts from those files when merging new versions! Understand that virtually nobody builds every single part of the kernel, not even the most common Linux distros like Ubuntu or Mint. This doesn’t mean you shouldn’t be taking these fixes because there ARE fixes for drivers you DO run. Take arm/arm64 and ext4 for example, which are the most common Android architecture and file system respectively. In 4.4, from 4.4.78 (version of the latest Oreo CAF tag) to 4.4.121 (latest upstream tag), these are the following numbers for the commits of those systems:

nathan@flashbox ~/kernels/linux-stable (master) $ git log --format=%h v4.4.78..v4.4.121 | wc -l2285 nathan@flashbox ~/kernels/linux-stable (master) $ git log --format=%h v4.4.78..v4.4.121 arch/arm | wc -l58 nathan@flashbox ~/kernels/linux-stable (master) $ git log --format=%h v4.4.78..v4.4.121 arch/arm64 | wc -l22 nathan@flashbox ~/kernels/linux-stable (master) $ git log --format=%h v4.4.78..v4.4.121 fs/ext4 | wc -l18

The most time-consuming part is the initial bring up; once you are all the way up to date, it takes no time at all to merge in a new release, which usually contains no more than 100 commits. The benefits that this brings (more stability and better security for your users) should necessitate this process though.

How to Merge Linux Stable Kernel into an Android Kernel

First you need to figure out what kernel version your Android device is running.

As trivial as this seems, it is necessary to know where you need to begin. Run the following command in your kernel tree:

make kernelversion

It will return back the version you’re on. The first two numbers will be used to figure out the branch you need (e.g. linux-4.4.y for any 4.4 kernel) and the last number will be used to determine what version you need to start with merging (e.g. if you are on 4.4.21, you’ll merge 4.4.22 next).

Grab the latest kernel source from kernel.org

kernel.org houses the latest kernel source in the linux-stable repository. At the bottom of that page, there will be three fetch links. In my experience, Google’s mirror tends to be the fastest but your results may vary. Run the following commands:

git remote add linux-stable https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable.gitgit fetch linux-stable

Decide if you want to merge the entire kernel or cherry-pick the commits

Next, you will need to choose if you want to merge the commits or cherry-pick. Here’s the pros and cons of each and when you may want to do them.

NOTE: If your kernel source is in the form of a tarball, you will most likely need to cherry-pick, otherwise you will get thousands of file conflicts because git is populating the history based purely on upstream, not what the OEM or CAF has changed. Just skip to step 4.

Cherry-picking:

Pros:

• Easier to resolve conflicts as you know exactly what conflict is causing an issue.
• Easier to rebase as each commit is on its own.
• Easier to bisect if running into issues

Cons:

• It takes longer as each commit has to be individually picked.
• Little more difficult to tell if commit is from upstream on first glance

Merge

Pros:

• It’s faster as you do not have to wait for all of the clean patches to merge.
• It’s easier to see when a commit is from upstream as you will not be the committer, the upstream maintainer will be.

Cons:

• Resolving conflicts can be a bit more difficult as you will need to look up which commit is causing the conflict using git log/git blame, it will not directly tell you.
• Rebasing is difficult as you cannot rebase a merge, it will offer to cherry-pick all of the commit individually. However, you should not be rebasing often, instead using git revert and git merge where possible.

I would recommend doing a cherry-pick to figure out any problem conflicts initially, doing a merge, then revert the problem commits afterwards so updating is easier (as merging is quicker after being up to date).

Add the commits to your source, one version at a time

The most important part of this process is the one version at a time part. There MAY be a problem patch in your upstream series, which could cause a problem with booting or break something like sound or charging (explained in the tips and tricks section). Doing incremental version changes is important for this reason, it’s easier to find an issue in 50 commits than upwards of 2000 commits for some versions. I would only recommend doing a full merge once you know all of the problem commits and conflict resolutions.

Cherry-picking

Format:

git cherry-pick <previous_tag>..<next_tag>

Example:

git cherry-pick v3.10.73..v3.10.74

Merge

Format:

git merge <next_tag>

Example:

git merge v3.10.74

I recommend keeping track of the conflicts in merge commits by removing the # markers.

How to Resolve Conflicts

We can’t give a step-by-step guide for resolving every single conflict, as it involves a good knowledge of C language, but here’s a few hints.

If you are merging, figure out what commit is causing the conflict. You can do this one of two ways:

1. git log -p v$(make kernelversion)..<latest_tag> to get the changes between your current version and the latest from upstream. The -p flag will give you the changes done by each commit so you can see.
2. Run git blame on the file to get the hashes of each commit in the area. You can then run git show –format=fuller to see if the committer was from mainline/stable, Google, or CodeAurora.

• Figure out if you already have the commit. Some vendors like Google or CAF will attempt to look upstream for critical bugs, like the Dirty COW fix, and their backports could conflict with upstream’s. You can run git log –grep=”<part_of_commit_message>” and see if it returns anything. If it does, you can skip the commit (if cherry-picking using git reset –hard && git cherry-pick –continue) or ignore the conflicts (remove the <<<<<< and everything between the ====== and >>>>>>).
• Figure out if there has been a backport that is messing up resolution. Google and CAF like to backport certain patches that stable wouldn’t. Stable will often need to adapt the resolution of the mainline commit to the abscence of certain patches that Google opts to backport. You can look at the mainline commit by running git show <hash> (the mainline hash will be available in the commit message of the stable commit). If there is a backport messing it up, you can either discard the changes or you can use the mainline version (which is what you will usually need to do).
• Read what the commit is trying to do and see if the problem is already fixed. Sometimes CAF may fix a bug independent of upstream, meaning you can either overwrite their fix for upstream’s or discard it, like above.

Otherwise, it may just be a result of a CAF/Google/OEM addition, in which case you just need to shuffle some things around.

Here is a mirror of the linux-stable kernel.org repository on GitHub, which can be easier for looking up commit lists and diffs for conflict resolution. I recommend going to the commit list view first and locating the problem commit to see the original diff to compare it to yours.

Example URL: https://github.com/nathanchance/linux-stable/commits/linux-3.10.y/arch/arm64/mm/mmu.c

You can also do it via the command line:

git log <current_version>..<version_being_added> <path>
git show <hash>

Solving resolutions is all about context. What you should ALWAYS do is make sure your final diff matches upstream’s by running the following commands in two separate windows:

git diff HEAD
git diff v$(make kernelversion)..$(git tag --sort=-taggerdate -l v$(make kernelversion | cut -d . -f 1,2)* | head -n1)

Enable rerere

Git has a feature called rerere (stands for Reuse Recorded Resolution), meaning that when it detects a conflict, it will record how you resolved it so you can reuse it later. This is especially helpful for both chronic rebasers with both merging and cherry-picking as you will just need to run git add . && git <merge|cherry-pick> –continue when redoing the upstream bringup as the conflict will be resolved how you previously resolved it.

It can be enabled by running the following command in your kernel repo:

git config rerere.enabled true

How to git bisect when running into a compiler or runtime error

Given that you will be adding a sizable number of commits, it’s very possible for you to introduce a compiler or runtime error. Instead of just giving up, you can use git’s built-in bisect tool to figure out the root cause of the issue! Ideally, you will be building and flashing every single kernel version as you add it so bisecting will take less time if needed but you can bisect 5000 commits without any issues.

What git bisect will do is take a range of commits, from where the issue is present to where it wasn’t present, and then start halving the commit range, allowing you to build and test and let it know if it is good or not. It will continue this until it spits out the commit causing your issue. At that point, you can either fix it or revert it.

1. Start bisecting: git bisect start
2. Label the current revision as bad: git bisect bad
3. Label a revision as good: git bisect good <sha1>
4. Build with the new revision
5. Based on the result (if the issue is present or not), tell git: git bisect good OR git bisect bad
6. Rinse and repeat steps 4-5 until the problem commit is found!
7. Revert or fix the problem commit.

NOTE: Mergers will need to temporarily run git rebase -i <good_sha> to apply all the patches to your branch for proper bisecting, as bisecting with the merges in place will often times checkout onto the upstream commits, meaning you have none of the Android specific commits. I can go into more depth on this upon request but trust me, it is needed. Once you have identified the problem commit, you can revert or rebase it into the merge.

Do NOT squash upstream updates

A lot of new developers are tempted to do this as it is “cleaner” and “easier” to manage. This is terrible for a few reasons:

• Authorship is lost. It’s unfair to other developers to have their credit stipped for their work.
• Bisecting is impossible. If you squash a series of commits and something is an issue in that series, it’s impossible to tell what commit caused an issue in a squash.
• Future cherry-picks are harder. If you need to rebase with a squashed series, it is difficult/impossible to tell where an conflict results from.

Subscribe to the Linux Kernel mailing list for timely updates

In order to get notified whenever there is an upstream update, subscribe to the linux-kernel-announce list. This will allow you to get an email every time a new kernel is released so you can update and push as quick as possible.