The Office of Information Technology is investigating networking issues affecting numerous services. More information will be reported as it becomes available.
Networking Issues
OIT Systems Maintenance, Thurs. July 9 from 3:00AM to 7:00AM
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 07/09/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Backup server upgrade
Description of change: OIT will upgrade on-prem crashplan backup server.
Downtime window start: 7/9/2020 5:00 AM
Downtime window end: 7/9/2020 6:00 AM
Affected services: Crashplan
Expected End-user impact: During this time, Crashplan backups will not work. Backups will resume once the upgrade is completed.
*************************************************************************************
Server Maintenance
Description of change: OIT will be enabling a feature to continually clean up old hostnames from DNS.
Downtime window start: 7/9/2020 5:00 AM
Downtime window end: 7/9/2020 5:00 AM
Affected services: Internal AD DNS (ad.fau.edu and irm.ad.fau.edu)
Expected End-user impact: Computers joined to the IRM or FAU domain may lose their DNS record if they have been unable to update it. This should self resolve in 24 hours or less or can be resolved manually.
*************************************************************************************
BHRIC VMware Horizon Upgrade to 7.12
Description of change: OIT will upgrade the BHRIC VMware Horizon to 7.12
Downtime window start: 7/9/2020 12:05 AM
Downtime window end: 7/9/2020 5:05 AM
Affected services: https://vhealth.fau.edu, https://vhealthon.fau.edu, horizon admin, vmware composer
Expected End-user impact: During this time, end users of the BHRIC VMware infrastructure will not be able to login to VMware services.
*************************************************************************************
BHRIC MySQL Server Storage Adjustment
Description of change: OIT will move the MySQL storage to an alternative storage partition to protect against back file growth and add certificates for mysql with tls.
Downtime window start: 7/9/2020 3:00 AM
Downtime window end: 7/9/2020 3:15 AM
Affected services: SaraAlert and Redcap
Expected End-user impact: During this time, SaraAlert and Redcap will be offline for 5 – 10 minutes while the data is moved.
*************************************************************************************
Web Services Load Balancer Maintenance
Description of change: OIT is performing an upgrade on load balancers that manage many of FAU’s major web services.
Downtime window start: 7/9/2020 3:00 AM
Downtime window end: 7/9/2020 7:00 AM
Affected services: All services served by virtual servers hosted by the BigIP load balancers. These include but may not be limited to: owlapp, sso, papercut, sas, myfau, banner, talisma, oud, www, filelocker, ds/ldap, dmz entry point for various coldfusion/grails/tomcat applications, tms, sctforms, etc.
Services that have cloud or NW presence will be failed over accordingly to lessen impact. E.g. www, oud, sso, myfau
Expected End-user impact: Users connecting to websites served by the BigIP load balancers may experience their sessions being disconnected intermittently as we proceed through the upgrade. Sites should not be down for extended periods of time as the units are being upgraded incrementally and there should be at least one unit providing services throughout the maintenance period.
*************************************************************************************
Global Web Services Load Balancer maintenance
Description of change: OIT is upgrading the Global Services Load Balancer with two newer ones. The Global Services Load Balancer (formerly Global Traffic Managers) are responsible for providing resiliency for major web sites/services that FAU provide. This change plan will account for the first step of adding the two new ones into the pool. A later change plan will be written to remove the two older ones.
Downtime window start: 7/9/2020 5:00 AM
Downtime window end: 7/9/2020 6:00 AM
Affected services: Web services/sites that are being managed by the BigIP DNS appliances (formerly GTM). These include many public FAU websites such as www.fau.edu, and departmental subdomains. May also include major services such as SSO, OUD, ADFS, wordpress.
Expected End-user impact: Durin this time, no end-user impact is expected. The new units have been configured to mirror what the old units serve up and their addition to the mix should be non-intrusive.
*************************************************************************************
[RESOLVED] Help Desk Ticketing System Unavailable
** UPDATE**
The help desk ticketing system is back online.
***********************************************
The help desk ticketing system is currently unavailable. We are working with the vendor to resolve the issue as soon as possible.
OIT
[RESOLVED] Several FAU applications/services unavailable
UPDATE: At this time all services have been restored.
********************************************************
We are currently experiencing issues with an Oracle back-end server. At this time, several applicatios/services are unavailable such as:
Student information in MyFAU
eGrades
Forms.fau.edu
University Appstatus
University Applications
OwlDone
We are working to resolve the issue as soon as possible.
OIT
[Resolved] Duo Two-Factor Authentication delays and failures
4/22/2020 8:16 AM: We were notified that the issues are resolved, and the 3rd-party network providers service issues have been resolved.
4/21/2020 4:15 PM: We have confirmed that the issues reported relate to 3rd-party network providers experiencing regional connectivity issues after a large fiber cut. Duo and Amazon are tracking with those providers to further understand when the cuts have been repaired or traffic is otherwise routable again, and are also investigating other potential solutions to route traffic at this time.
4/21/2020 1:35 PM: We have been notified that users may experience delays and failures when completing two-factor authentication. We will monitor the situation and update this notification as more information is received.
This may affect users logging into VPN, Workday, Banner, Student Self Service, MyFAU, Canvas, Email, and other services.
[RESOLVED]Office 365 email may be unavailable for some users
Microsoft resolved this issue by 2:40AM 4/21/20
——————————————————————————————————————
Office 365 users may experience 500 errors when attempting to access their email via web access, Mobile, or email clients.
Microsoft is currently investigating.
We will provide updates as they are available.
OIT
OIT Systems Maintenance, Thurs. April 16 from 3:00AM to 7:00AM
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 04/16/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Server Maintenance
Description of change: OIT will apply routine patches to redundant LDAP user/configuration information servers at disaster recovery location (NWRDC)
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: No service downtime expected but for full disclosure, the NWRDC oud servers are part of the cluster of: Production OUD (Oracle Unified Directory) servers, services that rely on them such as OIM (Identity Management), and servers/clients that rely on them for login authentication.
Expected End-user impact: During this time, no end-user impact is expected as any reboots necessary will be done in a rolling manner.
********************************************************
Server Upgrade
Description of change: OIT will be upgrading the software on its spam filtering appliances.
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: Inbound FAU Email, smtp.fau.edu
Expected End-user impact: During this time, inbound email may be delayed. Mail relayed through smtp.fau.edu may fail during the change window.
********************************************************
Server Maintenance
Description of change: OIT will add a test appliance to FAU’s web traffic director cluster to pull a copy of the configuration data
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: No service interruption expected but for full disclosure: a plethora of web domains and vanity domains are resolved via the Global Traffic Manager. Major ones are: www.fau.edu, sso.fau.edu, forms.fau.edu
Expected End-user impact: During this time, no end-user impact is expected.
********************************************************
Exit Form II
Description of change: OIT will apply a frevvo form for matriculated students who wish to withdraw/terminate from the university.
Downtime window start: 6:00 AM
Downtime window end: 8:00 AM
Affected services: Banner
Expected End-user impact: When students attempt to drop their last course in Banner they will now have an error message that redirects them to Exit Form II instead of Exit I.
********************************************************
Account cleanup
Description of change: OIT will be removing old user accounts
Downtime window start: 6:00 AM
Downtime window end: 6:15 AM
Affected services: AD login
Expected End-user impact: During this time, no end-user impact is expected
********************************************************
Server Maintenance
Description of change: We will be making a configuration change to Onbase webservers that will require a restart
Downtime window start 4/16/2020 3:00 AM
Downtime window end 4/16/2020 4:00 AM
OIT Units involved in the change (Internal Only) Systems and AS DEV
Affected services: Onbase web users will be logged off if connected during the restart (15 second window)
Expected End-user impact: During this time, no end-user impact is expected.
OIT Systems Maintenance, Thurs. April 9, from 3:00AM to 7:00AM
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 04/09/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Integration of new sexual harassment training tool
Description of change: OIT will integrate a new sexual harassment tool with Banner using a Snaplogic pipeline to integrate banner with the new tool from Everfi, a third party service.
Downtime window start: 12:00 AM
Downtime window end: 12:00 AM
Affected services: No services are affected.
Expected End-user impact: During this time no end-user impact is expected.
********************************************************
Server Upgrade
Description of change: OIT will be upgrading the software on one of its mail servers.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: On-premises exchange – mail.fau.edu
Expected End-user impact: During this time, no-end user impact is expected since all user mailboxes are hosted in Office 365. This change only impact on-premises exchange, which will be unavailable.
********************************************************
OIT Systems Maintenance, Thurs. April 2 from 3:00AM to 7:00AM
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 04/02/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Server Configuration Change
Description of change: OIT will be changing the domain several servers are joined to.
Downtime window start: 4:00 AM
Downtime window end: 6:00 AM
Affected services: NetApp services primarily used by the systems team. A development server used by the IEA team.
Expected End-user impact: During this time, services hosted on these systems will be unavailable.
********************************************************
WebFocus Server Maintenance
Description of change: OIT will be applying the Webfocus upgrade 8.2.06
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: WebFOCUS/Leap Ahead Production environment.
Expected End-user impact: During this time, services that listed above will be unavailable at times.
********************************************************
Server Upgrade
Description of change: OIT will be upgrading software on one of its servers.
Downtime window start: 3:00 AM
Downtime window end: 4:00 AM
Affected services: Azure AD and Active Directory synchronization
Expected End-user impact: During this time, new users and and user change syncs between Azure AD and On-Prem AD will be delayed while a full sync is done after the connector is upgraded. In previous upgrades the full sync is completed around noon of the same day.
********************************************************
Clockwork Maintenance
Description of change: OIT will be performing on the Clockwork system.
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: Not Applicable
Expected End-user impact: During this time, no end-user impact is anticipated.
********************************************************
OIT Systems Maintenance, Thursday, March 12 from 3:00AM to 7:00AM
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 03/12/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Database Maintenance
Description of change: OIT will apply January 2020 Oracle patchset updates to all production databases
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services:
Banner production
- Admin Pages (Forms)
- Application Navigator
- Banner Access Manager
- Banner self-service
- Touchnet – Payments/Deposits
- Workflow
- uAchieve – DARS
- FLVS
- Transcripts
Appworx
Harvest
ODS
FAUDW Warehouse
- DWPROD: OwlDone
- DWPROD: FinancialAid Attendance Survey
IEAP Warehouse
WebFOCUS/ReportCaster
OIM
- Account claiming
- Password resets
MSOP Instance
- ACH_PROD: ACH or VPAS
- GBP_PROD: Mandatory Course Entry Quiz
- WORKDESK_PROD: accounts.fau.edu – Accounts Self Service
- CANV_EGRADES_PROD: Canvas Electronic Grading “Banner Grade Submissions” in Canvas
- FAUVISITOR_PROD: Wireless network authentication for guests.
- FREVVOAPI_PROD: 4P Configuration, Admissions Status, Znumber page, ssoauth.fau.edu/*, egrades.fau.edu/* (except egrades itself)
- GROUPER_PROD: Group Management – grouper.fau.edu
- PHONEBILL_PROD: Phonebill application pre latest system.
- PWDEXPIRE_PROD: Password expiration
- REG_PROD: network registration, talon.fau.edu/netreg, talon.fau.edu/networkregistrationadmin, DHCP Logger
- FREVVO61_PROD: Frevvo
- NEWUNDERGRAD_PROD: Undergraduate Admissions
- OWLAPP_PROD: Graduate Admissions
- OWLRESIDENCY_PROD: Residency Project
- ONBOARDUSR: OwlDone
Expected End-user impact: During this time, the listed services will be temporarily offline while the patches are applied. Some applications may need to be restarted after the maintenance is complete.
*************************************************************************************
Server Maintenance
Description of change: OIT will be upgrading the components of several systems to resolve bugs and add support for newer hardware.
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services: All services
Expected End-user impact: During this time, no end-user impact is expected
********************************************************
Server Upgrade – Email
Description of change: OIT will be updating the software on the email spam filter appliances.
Downtime window start: 3:00 AM
Downtime window end: 4:00 AM
Affected services: Inbound mail, smtp.fau.edu
Expected End-user impact: During this time, sending mail through smtp.fau.edu might fail during the change window.
*************************************************************************************
Server Upgrade – Active Directory
Description of change: OIT will be upgrading the RDS licensing server to the latest version
Downtime window start: 3:00 AM
Downtime window end: 5:00 AM
Affected services: TS Licensing, RDS host
Expected End-user impact: During this time, adcon will be unavailable.
*************************************************************************************
VMWare vCenter upgrade
Description of change: OIT will upgrade the BHRIC vCentere to the latest version
Downtime window start: 3:00 AM
Downtime window end: 7:00 AM
Affected services: BHRIC Virtual Desktops (Horizon)
Expected End-user impact: During this time, users will not be able to start virtual desktops that are shutdown while the upgrade is in progress.
*************************************************************************************
Patch Forms Production Server
Description of change: OIT will update the Forms (Frevvo) Production Server to the latest version.
Downtime window start: 4:00 AM
Downtime window end: 6:00 AM
Affected services: Majority of the University online digital forms will be affected.
Expected End-user impact: During this time, affected forms will not be available for submission.
************************************************************************************
Non-Degree Residency Changes
Description of change: The residency process used for non-degree for undergrad and grad will be changed. Holds will be used to show status of residency process as well as OnBase forms and workflow to do verification of residency.
Downtime window start: 5:00 AM
Downtime window end: 6:00 AM
Affected services: Banner, Non-degree application
Expected End-user impact: During this time, no end-user impact is expected. This change impacts the way Registrar Residency officers process residency, and the way non-degree students submit their residency documentation.
*************************************************************************************
Remove med.ad.fau.edu domain
Description of change: OIT will remove the old med.ad.fau.edu domain.
Downtime window start: 3:00 AM
Downtime window end: 6:00 AM
Affected services: med.ad.fau.edu
Expected End-user impact: During this time, no end-user impact is expected as this domain is no longer in use.
*************************************************************************************