UPDATE: At this time all services have been restored.
********************************************************
We are currently experiencing issues with an Oracle back-end server. At this time, several applicatios/services are unavailable such as:
Student information in MyFAU
eGrades
Forms.fau.edu
University Appstatus
University Applications
OwlDone
We are working to resolve the issue as soon as possible.
OIT
4/22/2020 8:16 AM: We were notified that the issues are resolved, and the 3rd-party network providers service issues have been resolved.
4/21/2020 4:15 PM: We have confirmed that the issues reported relate to 3rd-party network providers experiencing regional connectivity issues after a large fiber cut. Duo and Amazon are tracking with those providers to further understand when the cuts have been repaired or traffic is otherwise routable again, and are also investigating other potential solutions to route traffic at this time.
4/21/2020 1:35 PM: We have been notified that users may experience delays and failures when completing two-factor authentication. We will monitor the situation and update this notification as more information is received.
This may affect users logging into VPN, Workday, Banner, Student Self Service, MyFAU, Canvas, Email, and other services.
Microsoft resolved this issue by 2:40AM 4/21/20
——————————————————————————————————————
Office 365 users may experience 500 errors when attempting to access their email via web access, Mobile, or email clients.
Microsoft is currently investigating.
We will provide updates as they are available.
OIT
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 04/16/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Server Maintenance
Description of change: OIT will apply routine patches to redundant LDAP user/configuration information servers at disaster recovery location (NWRDC)
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: No service downtime expected but for full disclosure, the NWRDC oud servers are part of the cluster of: Production OUD (Oracle Unified Directory) servers, services that rely on them such as OIM (Identity Management), and servers/clients that rely on them for login authentication.
Expected End-user impact: During this time, no end-user impact is expected as any reboots necessary will be done in a rolling manner.
********************************************************
Server Upgrade
Description of change: OIT will be upgrading the software on its spam filtering appliances.
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: Inbound FAU Email, smtp.fau.edu
Expected End-user impact: During this time, inbound email may be delayed. Mail relayed through smtp.fau.edu may fail during the change window.
********************************************************
Server Maintenance
Description of change: OIT will add a test appliance to FAU’s web traffic director cluster to pull a copy of the configuration data
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: No service interruption expected but for full disclosure: a plethora of web domains and vanity domains are resolved via the Global Traffic Manager. Major ones are: www.fau.edu, sso.fau.edu, forms.fau.edu
Expected End-user impact: During this time, no end-user impact is expected.
********************************************************
Exit Form II
Description of change: OIT will apply a frevvo form for matriculated students who wish to withdraw/terminate from the university.
Downtime window start: 6:00 AM
Downtime window end: 8:00 AM
Affected services: Banner
Expected End-user impact: When students attempt to drop their last course in Banner they will now have an error message that redirects them to Exit Form II instead of Exit I.
********************************************************
Account cleanup
Description of change: OIT will be removing old user accounts
Downtime window start: 6:00 AM
Downtime window end: 6:15 AM
Affected services: AD login
Expected End-user impact: During this time, no end-user impact is expected
********************************************************
Server Maintenance
Description of change: We will be making a configuration change to Onbase webservers that will require a restart
Downtime window start 4/16/2020 3:00 AM
Downtime window end 4/16/2020 4:00 AM
OIT Units involved in the change (Internal Only) Systems and AS DEV
Affected services: Onbase web users will be logged off if connected during the restart (15 second window)
Expected End-user impact: During this time, no end-user impact is expected.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 04/09/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Integration of new sexual harassment training tool
Description of change: OIT will integrate a new sexual harassment tool with Banner using a Snaplogic pipeline to integrate banner with the new tool from Everfi, a third party service.
Downtime window start: 12:00 AM
Downtime window end: 12:00 AM
Affected services: No services are affected.
Expected End-user impact: During this time no end-user impact is expected.
********************************************************
Server Upgrade
Description of change: OIT will be upgrading the software on one of its mail servers.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: On-premises exchange – mail.fau.edu
Expected End-user impact: During this time, no-end user impact is expected since all user mailboxes are hosted in Office 365. This change only impact on-premises exchange, which will be unavailable.
********************************************************
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 04/02/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Server Configuration Change
Description of change: OIT will be changing the domain several servers are joined to.
Downtime window start: 4:00 AM
Downtime window end: 6:00 AM
Affected services: NetApp services primarily used by the systems team. A development server used by the IEA team.
Expected End-user impact: During this time, services hosted on these systems will be unavailable.
********************************************************
WebFocus Server Maintenance
Description of change: OIT will be applying the Webfocus upgrade 8.2.06
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: WebFOCUS/Leap Ahead Production environment.
Expected End-user impact: During this time, services that listed above will be unavailable at times.
********************************************************
Server Upgrade
Description of change: OIT will be upgrading software on one of its servers.
Downtime window start: 3:00 AM
Downtime window end: 4:00 AM
Affected services: Azure AD and Active Directory synchronization
Expected End-user impact: During this time, new users and and user change syncs between Azure AD and On-Prem AD will be delayed while a full sync is done after the connector is upgraded. In previous upgrades the full sync is completed around noon of the same day.
********************************************************
Clockwork Maintenance
Description of change: OIT will be performing on the Clockwork system.
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: Not Applicable
Expected End-user impact: During this time, no end-user impact is anticipated.
********************************************************
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 03/12/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Database Maintenance
Description of change: OIT will apply January 2020 Oracle patchset updates to all production databases
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services:
Banner production
Appworx
Harvest
ODS
FAUDW Warehouse
IEAP Warehouse
WebFOCUS/ReportCaster
OIM
MSOP Instance
Expected End-user impact: During this time, the listed services will be temporarily offline while the patches are applied. Some applications may need to be restarted after the maintenance is complete.
*************************************************************************************
Server Maintenance
Description of change: OIT will be upgrading the components of several systems to resolve bugs and add support for newer hardware.
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services: All services
Expected End-user impact: During this time, no end-user impact is expected
********************************************************
Server Upgrade – Email
Description of change: OIT will be updating the software on the email spam filter appliances.
Downtime window start: 3:00 AM
Downtime window end: 4:00 AM
Affected services: Inbound mail, smtp.fau.edu
Expected End-user impact: During this time, sending mail through smtp.fau.edu might fail during the change window.
*************************************************************************************
Server Upgrade – Active Directory
Description of change: OIT will be upgrading the RDS licensing server to the latest version
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: TS Licensing, RDS host
Expected End-user impact: During this time, adcon will be unavailable.
*************************************************************************************
VMWare vCenter upgrade
Description of change: OIT will upgrade the BHRIC vCentere to the latest version
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services: BHRIC Virtual Desktops (Horizon)
Expected End-user impact: During this time, users will not be able to start virtual desktops that are shutdown while the upgrade is in progress.
*************************************************************************************
Patch Forms Production Server
Description of change: OIT will update the Forms (Frevvo) Production Server to the latest version.
Downtime window start: 4:00 AM
Downtime window end: 6:00 AM
Affected services: Majority of the University online digital forms will be affected.
Expected End-user impact: During this time, affected forms will not be available for submission.
************************************************************************************
Non-Degree Residency Changes
Description of change: The residency process used for non-degree for undergrad and grad will be changed. Holds will be used to show status of residency process as well as OnBase forms and workflow to do verification of residency.
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: Banner, Non-degree application
Expected End-user impact: During this time, no end-user impact is expected. This change impacts the way Registrar Residency officers process residency, and the way non-degree students submit their residency documentation.
*************************************************************************************
Remove med.ad.fau.edu domain
Description of change: OIT will remove the old med.ad.fau.edu domain.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: med.ad.fau.edu
Expected End-user impact: During this time, no end-user impact is expected as this domain is no longer in use.
*************************************************************************************
The University’s Forms Frevvo Google connector is offline for some forms. We are investigating the problem and will update this when the problem is resolved.
The University form server (frevvo) is back online and fully upgraded.
The University form server (frevvo) is currently offline. We are investigating the issue and will update this post regularly. This affects admissions applications, form resources, and other tools that relay on this service. If you need assistance please submit a ticket to the help desk.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 02/20/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Server configuration change – Storage
Description of change: OIT will be changing a parameter on it’s storage system to make inter-system communication more secure
Downtime window start: 6:00 AM
Downtime window end: 6:30 AM
Affected services: All Services
Expected End-user impact: During this time, no end-user impact is expected
*************************************************************************************
Server Configuration Change – Office365 Mail
Description of change: OIT will be increasing its spam checking for users on Office 365.
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: Office 365 mail
Expected End-user impact: During this time, users may see email that is classified as spam being sent to their junk email folder.
*************************************************************************************
Server Upgrade – DPM backup
Description of change
Upgrading data protection manager on boc22dpmus and nwdpmus from 2019 to 2019 UR1
Downtime window start: 2:00 AM
Downtime window end: 7:00 AM
Affected services: DPM backups of client desktops, secondary backups for scdpm.
Expected End-user impact: During this time, backup jobs on desktops with DPM installed will be suspended.
*************************************************************************************
Server Upgrade – Video Conferencing Servers
Description of change: OIT will be applying updates to virtualization hosts.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: Cisco meeting and videoconferencing services.
Expected End-user impact: During this time, Cisco meeting and video conferencing services will be unavailable.
*************************************************************************************
Server Migration – VIP Document management system
Description of change: OIT will be migrating a file server to a new one.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: VIP document management
Expected End-user impact: During this time, VIP document management system will be unavailable.
*************************************************************************************
Test Egrade posting process for Mid term grades.
Description of change: OIT is making sure mid term grades are bypassing Financial AID F/U/I grade validation logic
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: Not Applicable
Expected End-user impact: During this time, no end-user impact is expected.
********************************************************************************
Deleting disabled Single Sign On services
Description of change: OIT will be deleting the services that were disabled last week which did not cause disruptions.
Downtime window start: 12:00 AM
Downtime window end: 2:00 AM
Affected services:
TO DELETE
University Calendar Test, madcat1p 8090, madcat1p 8190, madcat1p, MadCatDr, IIS video0p, IIS VideoTech, IIS broadcast1, Ext Service Console Manager Development, LDAP Alias Manager Test, eGrades Scoreboard, eGrades Test, Starfish, Login, Canvas, Starfish Advising Software, FAUWeb1
FAUWeb4, Ibiweb 5, JIRA, MyFAU, MyFAU NW, PhoneBill, Precursor, RocketChat, RocketChat, SSOD, Undergraduate Application Production, FacSurv HTTPS, Undergraduate Application, FacSurv HTTPS
TO DELETE (DISABLED EXTRAS POST-ROLLUP)
OIMT – Match Resolution Manager (OIMT)
Athene – Athene ETD, OwlApps Password, PrivateEye 2.0, Athene Properties, Workday FAU API Prod
Athene Dev – Athene Dev ETD, Workday FAU API Dev
Athene Test – Athene Test ETD, Workday FAU API Test
Atlantic – Business Process, Online Forms, Frevvo API Prod, JIRA Reports, Library Archives Prod
Atlantic Test – Business Process Test, Grails Forms Test, Grails FAU Template, JIRA Reports Test, Library Archives Test
Banner NW – Banner SSO NW
Banner XE – Banner XE FAUP applicationNavigator, Banner Access Management Admin, Banner Access Management, Banner XE FAUP FacultyAttendanceTracking, Banner XE FAUP StudentAcademicHistory, Banner XE FAUP StudentAdvisorSSB, Banner XE FAUP StudentAttendanceTracking, Banner XE FAUP Student Attendance Tracking, Banner XE FAUP StudentClassSchedule, Banner XE FAUP StudentCourseCatalog, Banner XE FAUP StudentFacultyGradeEntry, Banner XE FAUP StudentOverall, Banner XE FAUP StudentRegistration, Banner XE FAUP StudentRegistrationSsb, Banner XE Student Self Service, Banner XE FAUP StudentSSB
Banner XE Dev – Banner XE DEVL applicationNavigator, Banner XE DEVL FacultyAttendanceTrackingSsb, Banner XE DEVL StudentAcademicHistory New, Banner XE DEVL StudentAdvisorSSB, Banner XE DEVL StudentAttendanceTracking, Banner XE DEVL StudentAttendanceTrackingSsb, Banner XE DEVL StudentClassSchedule, Banner XE DEVL StudentCourseCatalog, Banner XE DEVL StudentFacultyGradeEntry, Banner XE DEVL StudentOverall, Banner XE DEVL StudentRegistration, Banner XE DEVL StudentRegistrationSsb, Banner XE DEVL StudentSSB
Banner XE Test – Banner XE TEST applicationNavigator, Banner XE TEST FacultyAttendanceTracking, Banner XE TEST StudentAcademicHistory, Banner XE TEST StudentAdvisorSSB, Banner XE TEST StudentAttendanceTracking, Banner XE TEST Student Attendance Tracking, Banner XE TEST StudentClassSchedule, Banner XE TEST StudentCourseCatalog, Banner XE TEST StudentFacultyGradeEntry, Banner XE TEST StudentOverall, Banner XE TEST StudentRegistration, Banner XE TEST StudentRegistrationSsb, Banner XE TEST StudentSSB
AppDev Prod – AppDev Prod 01, AppDev Prod 02
AppDev Test – AppDev Test 01 8443, AppDev Test 02 8443
Onboarding – Onboarding Home, Onboarding Main
Chessboard – Chessboard Admin, Chessboard QA, Chessboard test, MFA QA
Short URLs – YOURLS Short URLs
Sympa – Sympa
Sympa NW – SympaNW
Owl Done Dev – Owl Done Dev, Owl Done Dashboard Dev, Owl Done Home Dev, Owl Done Main Dev
Owl App Test – OwlAppt Admin HTTPS
Owl Done – Owl Done, Owl Done ALEKS, Owl Done Dashboard, Owl Done Home, Owl Done Main
Self Service Dev – Self Service Development
Banner Test – Banner Test SSB
SSO Manager – SSO Manager HIST
Workflow Test – Workflow Test
Workflow Dev – Workflow Development
FAU Orientation – FAU Orientation, FAU Orientation Pre Prod
CAS – ClearPass, Ecollege, Ecollege4, ECollege, JSON Services Prod, LuminisSSO, CAS Testing, Services
CAS NW – CASNW
CAS Test – InCommon Test
Talon – ACH on Tal, Baseline Listener, Base Line Listen Test, Chessboard Production, EZproxy Test, EZproxy, Project Talon (Google), Multi Factor Enrollment, NetReg 2.0, Network Registration Client, Instructional Technologies Notify Tool, FAUNet Id Password Change Tool, PhoneBill 2.0, MyFAU SSO, Talon SwiftTickets
Tuna – Fin Aid Survey Dev, Library Archives Development
Expected End-user impact: During this time, no end-user impact is expected. There should be no impact as the services which will be deleted have already been disabled and are no longer in use.
*****************************************************************************************