RESOLVED – Banner Issues, Slowness and Timeouts

UPDATE 6/15/23 11:08AM:

The change rollback is now complete and we are seeing a significant positive performance impact on Banner Admin and other services. We believe the issue to be resolved at this moment, but we will still be monitoring Banner. If you are having issues, please close all your browser windows and sessions, and re-login to Banner Admin.

UPDATE 6/15/23 10:46AM:

Our Systems team is working on reverting back a change that was performed last night with some devices that host services such as Banner. Once this change is reverted, we will recheck the performance on Banner Admin, and see if it is back to normal.

UPDATE 6/15/23 10:00AM:

Ellucian is actively looking at the issue with the performance for Banner Admin. We see that at times it works correctly, but then will slow down and possibly time out. They have restarted services, which produced no change, they are now reviewing all the services in the environment to ensure everything is healthy. We apologize for the inconvenience, we will provide an update once we know more.

ORIGINAL MESSAGE:

We are experiencing issues with our Banner service right now. When users are in Banner Admin sometimes it is processing forms, tabs and saves slowly. In addition, we are seeing that saves and sometimes just form searches are producing a “Your session has expired. Click to reload and access login page” message.

We are having our Ellucian Managed Services Provider review the environment now. Once we have more information, we will update this message. For now users are going to get an inconsistent behavior from Banner Admin and possible Self Service, where it may operate correctly for a few minutes but then start to slow down.

Posted in Banner and Admin Systems

OIT Systems Maintenance, Thurs. June 15 from 12:01AM to 7:00AM

Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 06/15/2023 from 12:01 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:

Load balancer rotation
Description of change: OIT will be changing the active load balancer.
Downtime window start: 06/15/2023 6:00 AM
Downtime window end: 06/15/2023 7:00 AM
Affected services: Any websites behind the load balancers.
Main fau website, banner.fau.edu, links, lmsapps, talon, owlres, egrades, papercut, myowl, appstatus and more.
End-user impact: No end-user impact is expected.

Enable logging feature on authentication systems
Description of change: OIT will be enabling a new feature on some systems that provide authentication service.
Downtime window start: 06/15/2023 7:00 AM
Downtime window end: 06/15/2023 7:10 AM
Affected services: Active Directory
End-user impact: No end-user impact expected

Decommission system utilized by College IT
Description of change: OIT will be retiring a server utilized by College IT for management work
Downtime window start: 06/15/2023 7:00 AM
Downtime window end: 06/15/2023 7:10 AM
Affected services: remotecon
End-user impact: remotecon will no longer be available for use. College IT will be instructed to use adcon as a replacement

Enable 2factor authentication for system utilized by IT staff
Description of change: OIT will be enabling 2factor auth for a system utilized by IT staff
Downtime window start: 06/15/2023 7:00 AM
Downtime window end: 06/15/2023 7:10 AM
Affected services: adcon
End-user impact: IT staff will need to use DUO when using Remote Desktop to adcon

Course Entry Quiz: Enrollment modification
Description of change: OIT will make modifications to the Course Entry Quiz
Downtime window start: 06/15/2023 5:00 PM
Downtime window end: 06/15/2023 6:00 PM
Affected services: Course Entry Quiz
End-user impact: Students who have last attendance date in the system will not be required to take the quiz.

Posted in Banner and Admin Systems, Systems, Uncategorized

OIT System Maintenance, Thurs. June 8 from 12:01AM to 7:00AM

Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 06/08/2023 from 12:01 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:

Endpoint Security Change
Description of change: OIT will be implementing new safeguards for all endpoints.
Downtime window start: 06/08/2023 8:00 AM
Downtime window end: 06/08/2023 8:00 AM
Affected services: Domain joined Desktops and Laptops
End-user impact: No end-user impact is expected.

Configuration change for authentication system
Description of change: OIT will be making a configuration change to the systems that provide Windows authentication.
Downtime window start: 06/08/2023 5:00 AM
Downtime window end: 06/08/2023 5:05 AM
Affected services: Windows authentication
End-user impact: No end-user impact is expected.

BHRIC Domain Controller Maintenance
Description of change: Research Computing will be performing maintenance on the health.ad.fau.edu domain controllers.
Downtime window start: 06/08/2023 12:05 AM
Downtime window end: 06/08/2023 1:00 AM
Affected services: health domain
End-user impact: There should not be any impact to the end users during this change

Server Maintenance
Description of change: OIT will renew certs
Downtime window start: 06/08/2023 12:00 AM
Downtime window end: 06/08/2023 4:00 AM
Affected services: Any SSO Service
End-user impact: No end-user impact is expected.

Adding monitors to Global Traffic Manager
Description of change: OIT will be adding new monitors to the global traffic manager to make apps on certain clusters be more failover friendly.
Downtime window start: 06/08/2023 6:15 AM
Downtime window end: 06/08/2023 7:00 AM
Affected services: Services on the k8s cluster that are on the GTMs
End-user impact: No end-user impact is expected as no monitors will be removed until the new one returns green.

Load balancer rotation
Description of change: OIT will be changing the active load balancer.
Downtime window start: 06/08/2023 6:00 AM
Downtime window end: 06/08/2023 7:00 AM
Affected services: Any websites behind the load balancers, such as main fau website, banner.fau.edu, links, lmsapps, talon, owlres, egrades, papercut, myowl, appstatus and more.
End-user impact: We do not expect any end-user impact as the load balancer should fail over gracefully. In case there are issues, the sites behind the load balancers may not be accessible for a few seconds.

Cloud Native Upgrade
Description of change: The Office of Information Technology will be performing an upgrade to the Cloud Native Core in Boca.
Downtime window start: 06/08/2023 5:00 AM
Downtime window end: 06/08/2023 5:30 AM
Affected services: Management of Kubernetes clusters.
End-user impact: No end-user impact is expected.

Update Banner table
Description of change: OIT will make some changes in Banner to allow Registrar’s Office to updated holds.
Downtime window start: 06/08/2023 3:00 AM
Downtime window end: 06/08/2023 3:30 AM
Affected services: Banner
End-user impact: No end-user impact is expected

Emmersion Spanish Placement Test Deployment
Description of change: OIT will apply some changes to improve access to the Emmersion Spanish placement test
Downtime window start: 06/08/2023 5:00 PM
Downtime window end: 06/08/2023 6:00 PM
Affected services: OwlDocs
End-user impact: No end-user impace is expected.

Posted in Banner and Admin Systems, Email, Research Computing, Systems, Uncategorized

[RESOLVED 5:30 am] University Forms Server

During the University Form Server upgrade, the software experienced a critical error and the Office of Information Technology is working to restore services at this time. We suspect the system to be online shortly. We will update this post every 2 hours while we work to restore services. This affects University Admissions Applications, and numerous forms.

Posted in Banner and Admin Systems, Educational Technologies, Research Computing, Systems, Workday

[RESOLVED] TouchNet Systems Maintenance, Friday, June 9th from 3:00 AM to 7:00 AM

The TouchNet payment processing system will be undergoing maintenance -Starting from 3:00am to 7:00am (EST) Friday, June 9th. This affects student payment services (Bill+Payment), as well as all Marketplace Stores, uPay sites and TouchNet ready partner payment services. We expect all systems to be back online no later than 7:00 am (EST) that morning. When the system is back up and ready to go, we will notify you. You may wish to place warnings on your website, or take your store offline. If this scheduled downtime creates a conflict for your operations, please feel free to contact us at FIS@fau.edu for further discussion. Expected End-user impact: During this time all student payment services (Bill+Payment), as well as all Marketplace Stores, uPay sites and TouchNet ready partner payment services will be unavailable during this maintenance cycle.

Posted in Banner and Admin Systems, Workday

OIT Systems Maintenance, Thurs. June 1 from 12:01AM to 7:00AM

Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 06/01/2023 from 12:01 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:

Database migration
Description of change: OIT will be migrating several databases to different servers
Downtime window start: 06/01/2023 3:00 AM
Downtime window end: 06/01/2023 7:00 AM
Affected services: All databases involved in the migration will be unavailable during the window.
End-user impact: Users will not be able to access any services that are tied to the databases during the migration.

University Form Server Maintenance
Description of change: The University form server will undergo maintenance to improve performance and decrease the effort needed for upgrades.
Downtime window start: 06/01/2023 3:00 AM
Downtime window end: 06/01/2023 7:00 AM
Affected services: Forms.fau.edu, Apply.fau.edu, Cobapp.fau.edu, NonDegree.fau.edu
End-user impact: Users will not be able to submit forms during the maintenance.

Banner Server Maintenance
Description of change: Banner Production will need to be restarted to apply updates to the configurations. This is to ensure all settings are consistent with the Ellucian standards.
Downtime window start: 06/01/2023 5:00 AM
Downtime window end: 06/01/2023 5:20 AM
Affected services: Banner Admin, Banner Self Service.
End-user impact: Users will not be able to access Banner admin or Self-service during this time.

Banner SSOManager Maintenance
Description of change: Changes to the configuration of the Banner SSO Manager application will take place. This change will update settings within the application to ensure its proper functionality.
Downtime window start: 06/01/2023 6:00 AM
Downtime window end: 06/01/2023 7:00 AM
Affected services: SSOManager
End-user impact: No end-user impact is expected.

Posted in Banner and Admin Systems, Systems

OIT Systems Maintenance, Thurs. May 25 from 12:01AM to 7:00AM

Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 05/25/2023 from 12:01 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:

Server Maintenance
Description of change: OIT will be performing server maintenance on several core systems from 12:00am to 4:00am on May 25. During this time, the undergraduate application, MyOwl Services, Owl KPIs, and Owl Files will be unavailable.
Downtime window start: 05/25/2023 12:00 AM
Downtime window end: 05/25/2023 4:00 AM
Affected services: MyOwl Services, Owl KPIs,Owl Files, Undergraduate Application
End-user impact: The services listed above will be unavailabe during the maintenance window.

RDS Licensing Server upgrade
Description of change: Performing a Windows OS upgrade on our Remote Desktop Licensing Server
Downtime window start: 05/25/2023 12:00 AM
Downtime window end: 05/25/2023 6:00 AM
Affected services: This change will have an impact on any services using the server for remote desktop services, terminal services or VDI.
End-user impact: Faculty and Staff.

Server Maintenance
Description of change: OIT will renew certificates
Downtime window start: 05/25/2023 12:00 AM
Downtime window end: 05/25/2023 4:00 AM
Affected services: Any SSO Service
End-user impact: No end-user impact is expected

Server Maintenance
Description of change: OIT will be updating the software on one of its management servers.
Downtime window start: 05/25/2023 5:00 AM
Downtime window end: 05/25/2023 7:00 AM
Affected services: Advanced VPN clients
End-user impact: Advanced VPN clients will show that they are disconnected from their management server. This will not impact VPN/remote access.

OwlDone bug fixes and feature enhancment
Description of change: OIT will fix SOAR module for HCAA to have text displayed. Changed aid year to be dynamic to more accurately display status financial aid module. Converted owlsey image generation from coldfusion script to be natively handled within owldone.
Downtime window start: 05/25/2023 6:00 PM
Downtime window end: 05/25/2023 6:00 PM
Affected services: OwlDone
End-user impact: Students

BHRIC Maintenance
Description of change: Research Computing will be adding another domain controller in Boca and once We ensure that is synched up power off the older DC in Jupiter
Downtime window start: 05/26/2023 7:00 PM
Downtime window end: 05/28/2023 10:00 AM
Affected services: health.ad.fau.edu , https://vhealth.fau.edu, BHRIC File servers
End-user impact: No user impace should occur. However, if there is a problem this would prevent users from connecting into BHRIC VMWare.

Posted in Banner and Admin Systems, Systems, Uncategorized

OIT Systems Maintenance, Thurs. May 18 from 12:01AM to 7:00AM

Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 05/18/2023 from 12:01 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:

Disaster Recovery Testing
Description of change: OIT will perform disaster recovery testing on the wired and wireless network in preparation for hurricane season.
Downtime window start: 05/18/2023 6:00 AM
Downtime window end: 05/18/2023 7:00 AM
Affected services: Wired and Wireless network
End-user impact: All faculty, staff, and students.

RDS Licensing Server upgrade
Description of change: OIT will perform a Windows OS upgrade on our Remote Desktop Licensing Server
Downtime window start: 05/18/2023 12:01 AM
Downtime window end: 05/18/2023 6:00 AM
Affected services: This change will have an impact on any services using the server for remote desktop services, terminal services or VDI.
End-user impact: Faculty and Staff using the remote desktop services, terminal services or VDI.

Banner program change
Description of change: OIT will create a new Banner program to populate Workday payroll accounting adjustment records into Financial Aid College Work Study table
Downtime window start: 05/18/2023 12:01 AM
Downtime window end: 05/18/2023 1:00 AM
Affected services: Banner and Workday
End-user impact: No end-user impact is expected

Banner maintenance
Description of change: OIT will update Hold for Degree and Hold for Grades dates in Banner
Downtime window start: 05/18/2023 3:00 AM
Downtime window end: 05/18/2023 6:00 AM
Affected services: Banner
End-user impact: No end-user impact is expected during the change.

University Form Server Maintenance
Description of change: The University form server will undergo maintenance to improve performance and decrease the effort needed for upgrades.
Downtime window start: 05/18/2023 3:00 AM
Downtime window end: 05/18/2023 7:00 AM
Affected services: forms.fau.edu, apply.fau.edu, cobapp.fau.edu, nondegree.fau.edu
End-user impact: The services listed above may be unavailable during the maintenance window.

Service Upgrade
Description of change: OIT will upgrade some domain services at HBOI.
Downtime window start: 05/18/2023 4:00 AM
Downtime window end: 05/18/2023 4:30 AM
Affected services: Domain authentication on the HBOI campus
End-user impact: All users at the HBOI campus

Posted in Banner and Admin Systems, Network and Telecom, Research Computing, Systems, Uncategorized

[RESOLVED] Clean Address Application Service Interruption

The Clean Address issue below has been resolved.

Our service called clean address is currently experiencing issues. Any pages
that use this product to sanitize addresses are impacted. Two areas with most
use are:

The Personal Information Update Page:
https://myowl.fau.edu/contact/personalInformation/index)Banner

Admin SPAIDEN
screen, adding new users or updating addresses for existing users. Pages may not load correctly.

Posted in Banner and Admin Systems, Systems

OIT Systems Maintenance, Thurs. May 11 from 12:01AM to 7:00AM

Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 05/11/2023 from 12:01 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:

Service Upgrade
Description of change: OIT will upgrade domain services for the Harbor Branch campus.
Downtime window start: 05/11/2023 4:00 AM
Downtime window end: 05/11/2023 4:30 AM
Affected services: Domain authentication
End-user impact: All users at the HBOI campus.

Banner Maintenance
Description of change: OIT will be performing maintenance on Banner from 2:00 a.m. until 6:00 a.m. on May 11th. Banner will not be available during this time.
Downtime window start: 05/11/2023 2:00 AM
Downtime window end: 05/11/2023 6:00 AM
Affected services: Banner
End-user impact: All Banner systems will be unavailable during this time.

Posted in Banner and Admin Systems, Systems, Uncategorized