Application maintenance – University Form Server
Description of change: OIT will be applying improvements to the University Form Server login settings to improve performance.
Downtime window start: 6/16/2017 @ 5:00 AM
Downtime window end: 6/16/2017 @ 6:00 AM
Affected services: Forms. and Frevvo
Affected systems: frevvo1.fau.edu
Expected End-user impact: Users will not be able to login to https://forms.fau.edu/frevvo but most forms should continue to work without any issues.
The Single Sign On (SSO) maintenance has been completed successfully and Single Sign On is now online.
The Undergraduate Admissions application (apply.fau.edu) is back online.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 06/08/2017 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
Single Sign On (SSO) maintenance
Description of change: We will be updating the Single Sign On logout screen to include the following changes:
1. Remove Blackboard and Exchange WebMail links
2. Add Canvas link
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services: All Single Sign on systems may go offline including, Canvas, Blackboard, Workday, Google Applications, Office 365, and many more.
Affected systems: shibboleth1, shibboleth2, shibboleth3
Expected End-user impact: During the maintenance window, access to applications utilizing FAU NetID logins may be intermittently unavailable including Canvas, Blackboard, Workday, Google Applications, and Office 365
*************************************************************************************
Server maintenance
Description of change: OIT will be upgrading the VM version of several systems as well as expanding the processor capabilities for some VMs
Downtime window start: 3:00 AM
Downtime window end: 4:00 AM
Affected services: Labs print server, Appadmin, Canvas import tool, Networking DHCP server, PaperCut, Talisma, College of Medicine admissions website, AFTSS server, University Police – wireless lock control, High Performance Computing Primary DNS and Web Server, IDM, eGrades and BbGoogle, Starfish, Undergrad app, Middleware Identity Management
Affected systems: appadmin, as5, boc22canvascli, boc22cm102srv10, boc22cm102vrt101mrtg, boc22ora09, boc22pcapp01, boc22pclyprt01, boc22pcoffice01, boc22talweb2, comvmpsql1, comvmweb1, comvmwebdev1, controller, dsr2, dsr3, dsr5, duo, hpc, idm2, idm4p-oel6, madcat1p, pinky, shibboleth2, starfish, ugapp2, wildflymc1, wildflymc2, yakko
Expected End-user impact: During the maintenance window, access to services including eGrades, Talisma, Lab Printing, Undergrad admission applications, and College of Medicine admission applications will be intermittently unavailable.
*************************************************************************************
Network Registration maintenance
Description of change: The network registration system will be receiving an updated theme and be switched to a more reliable authentication mechanism.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: All services running on Talon.fau.edu including Chessboard, PrivateEye, Google Password Reset, Password Testing Tool, and others will be affected by this update.
Affected systems: talonx1, talonx2
Expected End-user impact: During the maintenance window, access to network registration and other services at the talon.fau.edu domain name may be intermittently unavailable.
*************************************************************************************
Boca Desktop Software Changes
Description of change: Alertus Desktop software will be installed on user computers.
Downtime window start: 3:30 AM
Downtime window end: 4:00 AM
Affected services: N/A
Affected systems: User PCs within FAU\BOC OU
Expected End-user impact: No end-user impact regarding service access is expected. New software will be pushed to computers on Boca campus and part of the FAU Active Directory Domain. As a result, there will be a new icon in the system tray. This icon will be for the new desktop alert system.
*************************************************************************************
Oracle database maintenance
Description of change: Applying the Oracle upgrade to FAUDW db on host odsprod
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: FAUDW production environment. Webfocus won’t be able to connect to the FAUDW database.
Affected systems: Odsprod
Expected End-user impact: During the maintenance window, access to Webfocus will be intermittent
*************************************************************************************
Oracle quarterly DB patches (odsprod)
Description of change: Applying Oracle PSU to DB’s on host odsprod
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: ODS and FAUDW production environment. Webfocus won’t be able to connect to the ODS or FAUDW databases
Affected systems: Odsprod
Expected End-user impact: Users will be unable to login to ODS or FAUDW during the period the database is offline. This should be no longer than 1 hour.
*************************************************************************************
Verasmart upgrade
Description of change: Upgrading Verasmart phone billing server to version 11.3
Downtime window start: 11:00 PM
Downtime window end: 12:00 AM
Affected services: Phone billing
Affected systems: Boc22comm1
Expected End-user impact: No end-user impact is expected.
*************************************************************************************
Identity Management Maintenance
Description of change: OIT will be performing maintenance on our Identity Management System.
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: OIM, OUD, MyFAU, Waveset
Affected systems: boc22oimp1, boc22oimp2
Expected End-user impact: Changes are being performed on services that Google Applications, MyFAU and password changes rely on. No end-user impact is expected.
*************************************************************************************
Server maintenance
Description of change: Maintenance will be performed on a redundant DNS server. DNS servers allow a user to access websites and services by using an easy-to-remember name
Downtime window start: 5:00 AM
Downtime window end: 5:30 AM
Affected services: External facing DNS
Affected systems: ns1
Expected End-user impact: No end-user impact is expected.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 06/01/2017 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
***************************************************
Google Applications Single Sign On (SSO) Update
When: 5:00AM – 6:00AM
Affected services: Google Applications
Affected systems: Google Applications, OwlApps
Description of maintenance: OIT will convert Google Applications authentication from CAS to Saml 2.0
User impact: Authentication to Google Applications may be interrupted during the conversion. Users should try to log in again after maintenance is completed.
***************************************************
Upgrade multiple systems – unavailable during upgrade
When: 3:00AM – 4:00AM
Affected services: Undergrad Application, SACS, Remote Management, College of Science web, Networking What’s up, Networking napalm, Shoutcast, Labs XenApp, vCenter, Middleware monitoring system, Middleware monitoring/management system, User Services account management tool
Affected systems: boc22cm102srv1, boc22vcenter1, boc22whatsup, chessboard, cosweb1
cuttlefish, dsr6, eprint2, jenkins2, lintalon1, mcclient, napalm, piwik1, remotecon, s165n113, sacs, sepiidae, shoutcast, tuna2p, ugapp3, ugapp4, XenAPP00, XenAPP01, XenAPP02
Description of maintenance: OIT will be upgrading the VM version of several systems as well as expanding the processor capabilities for some VMs
User impact: The listed services may be unavailable for approximately 5 minutes during the window
***************************************************
Recreate dnsjup2 virtual machine
When: 4:00AM – 5:00AM
Affected services: DNS
Affected systems: dnsjup2 [131.91.213.91]
Description of maintenance: OIT is deleting and recreating the dnsjup2 VM using the same virtual hard disk. This is being done because this VM shares the same VM id as dnsboc1 and is causing various issues with SCVMM managing the clusters they reside on as well as the VMs themselves.
User impact: Any systems that rely on dnsjup2 for name resolution will fail during the change window.
***************************************************
Replace NS with new Linux VM
When: 4:30AM – 5:00AM
Affected services: External facing DNS
Affected systems: ns
Description of maintenance: OIT is replacing an old Solaris branded computer server that is functioning as ns (one of three external facing domain name servers) with a new VM that is running the Linux operating system.
User impact: No expected end-user impact
***************************************************
Appworx Production Upgrade
When: 5:00AM – 6:00AM
Affected services: Appworx
Affected systems: enzo, bannerprd
Description of maintenance: OIT is upgrading Appworx production to latest release
User impact: Appworx and Appworx jobs will be unavailable during this time
***************************************************
Blackboard Restarts
When: 5:00AM – 6:00AM
Affected services: Blackboard
Affected systems: Blackboard
Description of maintenance: OIT is restarting blackboard application servers to fix an analysis issue that some professors may be having.
To start with the ramp down of blackboard we will only be bringing up 4 user facing app servers.
User impact: Blackboard will be intermittently unavailable between 5 and 6 AM
***************************************************
We are currently experiencing issues with single-sign on. The user will be presented with an error message when attempting to sign in to services such as Workday, Canvas, Helpdesk, Blackboard, and any other application that is utilizing single-sign on. Our team is working on restoring services as soon as possible.
We apologize for the inconvenience this may cause.
OIT
Google is currently investigating a phishing email that is appearing as Google Docs. Please do not click on the link for any unknown shared google doc requests. If you have clicked the link, please make sure to change your Owl Apps and FAU password as soon as possible.
For further assistance resetting your password or any questions please contact the OIT help desk at 561-297-3999.
OIT
**** UPDATE 8:46AM *****
The issue with single-sign on has been resolved and all services have been restored at this time. You may need to close all open browser windows and open a new session before attempting to login.
We thank you for your patience during this time.
OIT
*************************************************************************************************************
We are currently experiencing issues with single-sign on. The user will be presented with an error message when attempting to sign in to services such as Workday, Canvas, Helpdesk, Blackboard, and any other application that is utilizing single-sign on. Our team is working on restoring services as soon as possible.
We apologize for the inconvenience this may cause.
OIT
Issue has been resolved.
———————————————————
Some users may experience an error trying to log into blackboard. We are currently investigating and working to resolve as soon as possible.
We apologize for the inconvenience.
OIT
Amazon has verified that uploads to their service should be working again; users should be seeing improved performance with their uploads to Canvas. Our DevOps team is continuing to monitor the situation, but we are not currently aware of any lingering issues that affect Canvas functionality at this time.
In our previous update, we mentioned there would still be areas of impaired functionality between Canvas and Amazon. The biggest area of impact right now is that uploads are not yet working. This includes student uploads to assignments, instructor grade uploads, and similar functions, but also the ability for Canvas’ background processes to upload files such as admin reports (which is required as part of the process to generate a report at the account level). You may continue to see issues with this, and other areas in Canvas, as Amazon works to fully restore all services.
Canvas performance and service recovery continues to progress quickly. Although many users should now be able to access Canvas, there may still be areas of impaired functionality as we work through remaining issues.
We are beginning to see positive indications of recovery and have successfully tested workflows that were previously failing. We are still awaiting full resolution, and we will provide updates as the situation continues to improve.
AWS is still working through their recovery process. Unfortunately, the number of Amazon services that have been impacted has grown in the time it took to find the root cause, and it will be a significant effort on their side to recover all of the services. They are understandably starting with the most critical ones. Since Canvas depends on so many of their services, a full recovery may still take some time.
On our side, our DevOps team has moved on to other ideas about how to get from a “service disruption” state to a “degraded performance” state in Canvas. We are also discussing the plans for addressing similar circumstances in the future, though our options are limited due to the perniciousness of this incident; but we are considering all options at this time.
Amazon is continuing to work through their recovery process. On our side, our DevOps team has implemented a temporary change to ensure tools and apps not hosted on AWS (Amazon Web Services) are still accessible to those that are able to access Canvas, which is an improvement to the complete service disruption we have had since 10:37 AM MST. However, the majority of Canvas users are still unable to access their Canvas site, due to the outage with AWS.
We will continue our efforts to ensure a good experience with Canvas for users once they are able to access the site again, and will provide an update on the overall issue within the next 30 minutes.
As Amazon works to restore availability in their systems, our DevOps team continues their efforts to expedite the process to restore access to Canvas. We will provide a new update on their progress in 30 minutes or less.
Amazon Web Services has informed us that they have identified the underlying root cause of the issue and they are beginning the remediation process. Our internal DevOps team continues to explore options to facilitate faster recovery.
Amazon is still working to restore server access for sites that have been affected by their outage today, including many Canvas sites. They will keep us updated on their progress.
Amazon has narrowed the scope of their investigation and has identified a specific region impacted by the networking issue. They are actively working on a solution. Our own DevOps team is investigating options that may allow us to work around the problem. We will provide another update in 15 minutes.
Amazon Web Services is currently experiencing what appears to be a large-scale networking issue that has impacted Instructure along with many other companies. We are working with Amazon to diagnose the problem and waiting for updates on their mitigation timeline. We will keep you posted as soon as we have more information.
Updates will follow as they become available.