Deep Dive: JunOS Upgrade best practices
Upgrades of network devices can be like open-heart surgery. It is a risky procedure that requires meticulous planning and precise execution. Let us help you to make your upgrade experience hassle-free.
#Juniper Networks #Network as a Service #Deep Dive
Junos OS is a single network operating system running on an underlying FreeBSD, that is used on routers, switches, and firewalls in order to simplify network operations, administration, and maintenance across Juniper products.
The reason for upgrading Junos OS on a regular basis might vary from out-of-compliance software, to deploying new features, or standardizing on a corporate version on all routers/switches. As most of the companies are relying on their network for their business, an up to date software brings benefits that are worth the investment. In this article we will discuss about the dilemma between upgrading or staying or a stable Junos OS release, we will offer a brief explanation of Junos OS package naming convention, useful links and tools that help finding the right release and never the less a general method of procedure.
Upgrading or staying on a stable Junos release?
- When you are dealing with this dilemma, there are some questions that can help to make a decision, for example:
When is the current Junos OS version End of Life/End of Support?
Make sure you check: Junos OS Dates & Milestones - Does your network require new features?
To compare new features, check the following link: https://apps.juniper.net/feature-explorer/ - Is the current Junos OS version vulnerable to known issues that could affect your network?
PR checks can be made using Problem Report Search.
Junos Releases explained
The software release numbers of Junos are in a major and minor format. The major number is the version of Junos for a particular calendar year and the minor release indicates which quarter the software was released.
Each release of Junos is supported for 18 months. The last release of Junos in the calendar year is known as the Extended End of Life (EEOL), and this release is supported for 36 months.
Example:
- 20.4 is originally from 2020 Q4 and 21.1 is from 2021 Q1
- Junos 20.1 to 20.2 would introduce new features, and Junos 20.1R1 to 20.1R2 would introduce bug fixes.
The advantage of using the last Junos release of the previous calendar year (EEOL release) is that for example 20.1 will stop providing bug fixes after 18 months, while 20.4 will continue to have bug fixes for 36 months.
There are also a couple of different types of Junos that are released more frequently to resolve issues: maintenance and service releases.
Maintenance releases fix a collection of issues and they are prefixed with “R” while Service Releases are released on demand to specifically fix a critical issue that has yet to be addressed by a maintenance release. The releases are prefixed with “S”.
Example of the Software Package Naming Conventions can be seen below:
junos-install-mx-x86-64-21.2R3-S4.3.tgz
junos-install-mx – the name of the software package
x86-64 – architecture-bit value x86-64 indicates that the software package is for 64-bit Junos OS and supports the x86 architecture
21.2 – major software release
R3 – maintenance release
S4 – Service release software. SR represents the 4th service release on top of 21.2R3
3 – the revision number
JUNOS Service Release software – On the JUNOS Software download page when you open the JUNOS image type dropdown menu there are two options: “Junos” and “Junos Service Release” .
Selecting “Junos” will bring you to Junos maintenance releases while “Junos Service Release” brings you to the “Service Releases” download page.
Screen capture of Juniper Software Downloads page
Finding the right Release
In order to help customers select a Junos software version that aligns with their deployment needs, Juniper offers various help: Junos Software Versions – Suggested Releases to Consider and Evaluate
For a complete analysis of your environment, you can choose “PIIR Services” (Product Issue Impact Review Services). This of course will involve additional charges from Juniper. Our team can offer a similar service as well, for this please contact info@umb.ch.
Additionally, Juniper offers a tool called Juniper Software Support Evaluation Tool (JSSET). This tool helps you to view the latest proactive bug notifications. You can build customized queries of Junos OS software issues based on your network profile (Junos version and hardware inventory), and it helps to assess the service impact of each problem report (PR). Preview query results in the tool to make an impact assessment, and export bug information to an Excel spreadsheet for offline analysis and action. Check JSSET Quick Start Guide for understanding how the tool works.
Screen capture of Juniper Software Support Evaluation Tool (JSSET)
Bug-Bounty
Sometimes the upgrade decision must be taken because in your current Junos version, you might have hit a bug or a known PR and the only solution is a software upgrade.
In order to find what is new in your Junos release and also known or fixed issues, check the release notes of your desired version.
Once you have decided upon a Junos version that you would like to install, a very common question comes to mind. What is the recommended upgrade path?
Upgrade Path
The general rule is that you should not skip more than two consecutive major releases.
In the example below, we consider the upgrade from 18.3 to 20.3R3 and a patch update from 18.2R2.6 to 18.2R3-S8:
From 18.1 to 20.3R3, there are the following major versions in between: 18.1.R3 – 18.2 – 18.3 – 18.4 – 19.1 – 19.2 – 19.3 – 19.4 – 20.1 – 20.2 – 20.3.R3
So the correct upgrade path would be: 18.1R3 → 18.4 → 19.3 → 20.2 → 20.3R3
For a patch update from 18.2R2.6 to 18.2R3-S8, a direct path can be followed since this would not skip any major releases.
Upgrade Planning
To make an upgrade successful and to avoid adverse impacts, the upgrade must be carefully planned and executed. This requires a Method of Procedure (MOP) created with a “Plan B” in case of a failure.
Some things to consider while preparing your procedure are the following:
A. Preliminary Activity
1. Download the new Junos version from juniper.net – Support – Downloads
2. Save the current device configuration (in case of a failure after a software upgrade).
> file list /config/
Sftp: > get /config/juniper.conf.gz
Or:
> show configuration | display set | no-more
3. Check if enough space is available on your device:
> show system storage
If there is not enough space, do a storage clean up:
> request system storage cleanup
> yes
To delete a specific file (ex. old Junos package):
> file delete /var/tmp/jinstal..
4. Upload the new Junos OS Image on the device.
5. In order to set yourself up for success and be able to spot an issue after a software upgrade, you must begin with a known reference point.
Below are some pre and post validation commands:
System health
> show chassis alarms | no-more
> show system core-dumps | no-more
> show pfe statistics traffic | match drop
> show pfe statistics error | no-more
> show system processes extensive | no-more
> show system uptime no-forwarding | no-more
> show chassis fpc detail | no-more
> show chassis environment | no-more
> show chassis routing-engine | no-more
Infrastructure service checks
> show interfaces descriptions | match down | no-more
> show interfaces | match “Physical|rate” | no-more
> show isis adjacency | no-more
> show ldp session | no-more
> show bfd session | no-more
> show route summary | no-more
> show bgp summary | no-more | match Establish
B. Upgrade Activity:
1. Login to the device and enable terminal monitor
> monitor start internal-error | except md5 | except repeated
Single routing engine:
Make a snapshot of the current version
> request system snapshot
Initiate the Upgrade:
> request system software add reboot /var/tmp/junos-package-name-architecture-bit-release-Cx.y.tgz
Dual routing engine:
Make a snapshot of the current version
> request system snapshot routing-engine both
Deactivate all NSR functions:
If graceful Routing Engine switchover (GRES) or nonstop active routing (NSR) is enabled when you initiate a software installation, the software does not install properly.
Make sure you deactivate GRES (if it is enabled). By default, NSR is disabled. If NSR is enabled, remove the nonstop-routing statement from the [edit routing-options] hierarchy level to disable it.
# deactivate chassis redundancy graceful-switchover
# deactivate routing-options nonstop-routing
Upgrade Backup RE:
> request routing-engine login other-routing-engine
> request system software add reboot /var/tmp/junos-package-name-architecture-bit-release-Cx.y.tgz
RE Switchover:
> request chassis routing-engine master switch
Upgrade Primary RE:
> request routing-engine login other-routing-engine
> request system software add reboot /var/tmp/junos-package-name-architecture-bit-release-Cx.y.tgz
Enable Redundancy – activate all NSR functions
# activate chassis redundancy graceful-switchover
# activate routing-options nonstop-routing
Check if all protocols are in sync
> show task replication
Trigger final RE switchover with NSR
> request chassis routing-engine master switch
2. Commit Full
> commit full
3. Run post-upgrade snapshot
> request system snapshot
4. After reboot, check if the router is running the new software version and make sure it is alarm-free.
To confirm that everything works as expected, make again the System health and Infrastructure service checks and compare them with the initial outputs.
C. Roll-back activity:
“Hope for the best, but prepare for the worst”.
Prepare before the activity a plan to restore the correct functionality of the node in case of failure.
Even if this is not so common, sometimes Field Service might be required (if you lost connection and need console access), make sure you have a Field Engineer available during your activity to go on site if necessary.
Example 1:
The device has booted from the Backup Junos OS image
Usually a reboot can fix this problem, but if this doesn`t help:
Example 2:
Junos OS Upgrade failure/crash – the system is stuck on boot up
Solution: https://kb.juniper.net/InfoCenter/index?page=content&id=KB26175
Conclusion
The world we live in today is totally dependent on network infrastructure, from having the possibility to make an emergency call, to reading and relaxing on social media, or fulfilling work responsibilities.
To keep up with the growing data demands, clinging to the old infrastructure is not safe or practical. Growth and productivity in any organization require a reliable network.
Additional information can be found on the link below:
