The Office of Information Technology is investigating networking issues affecting numerous services. More information will be reported as it becomes available.
Networking Issues
OIT Systems Maintenance, Thurs. July 9 from 3:00AM to 7:00AM
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 07/09/2020 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
*************************************************************************************
Backup server upgrade
Description of change: OIT will upgrade on-prem crashplan backup server.
Downtime window start: 7/9/2020 5:00 AM
Downtime window end: 7/9/2020 6:00 AM
Affected services: Crashplan
Expected End-user impact: During this time, Crashplan backups will not work. Backups will resume once the upgrade is completed.
*************************************************************************************
Server Maintenance
Description of change: OIT will be enabling a feature to continually clean up old hostnames from DNS.
Downtime window start: 7/9/2020 5:00 AM
Downtime window end: 7/9/2020 5:00 AM
Affected services: Internal AD DNS (ad.fau.edu and irm.ad.fau.edu)
Expected End-user impact: Computers joined to the IRM or FAU domain may lose their DNS record if they have been unable to update it. This should self resolve in 24 hours or less or can be resolved manually.
*************************************************************************************
BHRIC VMware Horizon Upgrade to 7.12
Description of change: OIT will upgrade the BHRIC VMware Horizon to 7.12
Downtime window start: 7/9/2020 12:05 AM
Downtime window end: 7/9/2020 5:05 AM
Affected services: https://vhealth.fau.edu, https://vhealthon.fau.edu, horizon admin, vmware composer
Expected End-user impact: During this time, end users of the BHRIC VMware infrastructure will not be able to login to VMware services.
*************************************************************************************
BHRIC MySQL Server Storage Adjustment
Description of change: OIT will move the MySQL storage to an alternative storage partition to protect against back file growth and add certificates for mysql with tls.
Downtime window start: 7/9/2020 3:00 AM
Downtime window end: 7/9/2020 3:15 AM
Affected services: SaraAlert and Redcap
Expected End-user impact: During this time, SaraAlert and Redcap will be offline for 5 – 10 minutes while the data is moved.
*************************************************************************************
Web Services Load Balancer Maintenance
Description of change: OIT is performing an upgrade on load balancers that manage many of FAU’s major web services.
Downtime window start: 7/9/2020 3:00 AM
Downtime window end: 7/9/2020 7:00 AM
Affected services: All services served by virtual servers hosted by the BigIP load balancers. These include but may not be limited to: owlapp, sso, papercut, sas, myfau, banner, talisma, oud, www, filelocker, ds/ldap, dmz entry point for various coldfusion/grails/tomcat applications, tms, sctforms, etc.
Services that have cloud or NW presence will be failed over accordingly to lessen impact. E.g. www, oud, sso, myfau
Expected End-user impact: Users connecting to websites served by the BigIP load balancers may experience their sessions being disconnected intermittently as we proceed through the upgrade. Sites should not be down for extended periods of time as the units are being upgraded incrementally and there should be at least one unit providing services throughout the maintenance period.
*************************************************************************************
Global Web Services Load Balancer maintenance
Description of change: OIT is upgrading the Global Services Load Balancer with two newer ones. The Global Services Load Balancer (formerly Global Traffic Managers) are responsible for providing resiliency for major web sites/services that FAU provide. This change plan will account for the first step of adding the two new ones into the pool. A later change plan will be written to remove the two older ones.
Downtime window start: 7/9/2020 5:00 AM
Downtime window end: 7/9/2020 6:00 AM
Affected services: Web services/sites that are being managed by the BigIP DNS appliances (formerly GTM). These include many public FAU websites such as www.fau.edu, and departmental subdomains. May also include major services such as SSO, OUD, ADFS, wordpress.
Expected End-user impact: Durin this time, no end-user impact is expected. The new units have been configured to mirror what the old units serve up and their addition to the mix should be non-intrusive.
*************************************************************************************
[RESOLVED] TouchNet Systems Maintenance, Friday, June 26th from 3:00AM to 7:00AM
The TouchNet payment processing system will be undergoing maintenance -Starting from 3:00am to 7:00am (EST) Friday, 6/26/2020. This affects student payment services (Bill+Payment), as well as all Marketplace Stores, uPay sites and TouchNet ready partner payment services. We expect all systems to be back online no later than 7:00 am (EST) that morning. When the system is back up and ready to go, we will notify you. You may wish to place warnings on your website, or take your store offline. If this scheduled downtime creates a conflict for your operations, please feel free to contact us at FIS@fau.edu for further discussion. Expected End-user impact: During this time all student payment services (Bill+Payment), as well as all Marketplace Stores, uPay sites and TouchNet ready partner payment services will be unavailable during this maintenance cycle.
Posted in Banner and Admin Systems, Workday
[RESOLVED] Workday Monthly Maintenance – June 12-13, 2020
Beginning at June 12, 2020, 11:00 PM Pacific time (June 13, 2020, 2:00 AM Eastern time), Workday Production tenants will be unavailable for a period of 8 hours until June 13, 2020, 7:00 AM Pacific time (June 13, 2020, 10:00 AM Eastern time).
In addition, the Implementation, Sandbox, and Sandbox preview tenants will be unavailable for 12 hours beginning on June 12, 2020, 6:00 PM Pacific time (June 12, 2020, 9:00 PM Eastern time), until June 13, 2020, 6:00 AM Pacific time (June 13, 2020, 9:00 AM Eastern time).
Posted in Workday
[RESOLVED] Help Desk Ticketing System Unavailable
** UPDATE**
The help desk ticketing system is back online.
***********************************************
The help desk ticketing system is currently unavailable. We are working with the vendor to resolve the issue as soon as possible.
OIT
[RESOLVED] Several FAU applications/services unavailable
UPDATE: At this time all services have been restored.
********************************************************
We are currently experiencing issues with an Oracle back-end server. At this time, several applicatios/services are unavailable such as:
Student information in MyFAU
eGrades
Forms.fau.edu
University Appstatus
University Applications
OwlDone
We are working to resolve the issue as soon as possible.
OIT
[Resolved] Duo Two-Factor Authentication delays and failures
4/22/2020 8:16 AM: We were notified that the issues are resolved, and the 3rd-party network providers service issues have been resolved.
4/21/2020 4:15 PM: We have confirmed that the issues reported relate to 3rd-party network providers experiencing regional connectivity issues after a large fiber cut. Duo and Amazon are tracking with those providers to further understand when the cuts have been repaired or traffic is otherwise routable again, and are also investigating other potential solutions to route traffic at this time.
4/21/2020 1:35 PM: We have been notified that users may experience delays and failures when completing two-factor authentication. We will monitor the situation and update this notification as more information is received.
This may affect users logging into VPN, Workday, Banner, Student Self Service, MyFAU, Canvas, Email, and other services.
[RESOLVED] TouchNet Systems Maintenance, Friday, April 24th from 3:00AM to 7:00AM
The TouchNet payment processing system will be undergoing maintenance -Starting from 3:00am to 7:00am (EST) Friday, 4/24/2020. This affects student payment services (Bill+Payment), as well as all Marketplace Stores, uPay sites and TouchNet ready partner payment services. We expect all systems to be back online no later than 7:00 am (EST) that morning. When the system is back up and ready to go, we will notify you. You may wish to place warnings on your website, or take your store offline. If this scheduled downtime creates a conflict for your operations, please feel free to contact us at FIS@fau.edu for further discussion. Expected End-user impact: During this time all student payment services (Bill+Payment), as well as all Marketplace Stores, uPay sites and TouchNet ready partner payment services will be unavailable during this maintenance cycle.
Posted in Banner and Admin Systems, Workday
Forms restart
The forms service forms.fau.edu (frevvo) needed to be restarted due to a newly discovered issue. We are investigating the problem and will restart the service as needed until the vendor resolves the issue.
University Form Server indexing issue
We have discovered a problem with the University form server indexing newly submitted documents. While this doesn’t affect reception and data integrity it is causing delays in processing some form posts that require manual interaction. We are going to attempt several adjustments at 10 pm in an attempt to resolve this issue. Frevvo will be offline for 5 to 10 minutes while it is restarted.