This re-indexing operation is now complete and the Echo web interface is responsive once again.
Original post:
The Echo360 class video system is currently undergoing a resource re-indexing operation which is causing it’s web interface to respond slowly.
This process is expected to complete by 11pm tonight and at that time the web interface will respond normally again.
The network maintenance scheduled for Thursday, December 19th has been completed.
Thank you
Services down: idm gateway, sharepoint, dns resolutions, talapp, ldap and others that are located on the cluster. Systems is troubleshooting.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 12/19/2013 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
When: 3:00AM – 4:00AM
Affected services: WordPress
Affected systems: WordPress
Description of maintenance: We are doing updates to enhance wordpress functionality
User impact: No end user impact is expected
When: Thursday, December 19, 2013: 3:00-5:00 AM, (Friday December 20, 2013 5:00AM-7:00AM if necessary).
Affected services: Blackboard
Description: Blackboard has released an update to several Building Blocks including SafeAssign, Collaborate, Blackboard Mobile, Cloud Profiles, Retention Center, and Content Editor. Additionally, publishers Pearson, McGraw Hill and Cengage have released updated Building Blocks. We will also be updating the Advanced User Search Building Block, and installing two new Building Blocks, My Surveys and Quickly. We will get as many of these done as we can during the time frame on Thursday. Any remaining Building Blocks will be updated on Friday morning. We do not expect any downtime.
When: 5:30AM – 6:00AM
Affected services: Network
Affected systems: Network
Description of maintenance: The goal of this change is to enable new network functionality for FAU. OIT will perform required changes to the network infrastructure.
User impact: No end user impact is expected.
When: 5:00AM – 6:00AM
Affected services: Network
Affected systems: Cisco Nexus core router
Description of maintenance: The goal of this change is to upgrade the software on the Cisco Nexus core router.
User impact: This change will result in a brief network outage as the upgrade is completed. All devices on the FAU network should reconnect after the upgrade and reboot has been completed.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Thursday, 12/12/2013 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
When: 3:00AM – 7:00AM
Affected services: Banner Production
Affected systems: Banner Production
Description of maintenance: The following modules will be installed to the specified version in production FAUP: Finance Self Service 8.6.0.2 patch and Finance 8.9.0 upgrade
User impact: There will be no end user impact.
When:5:00AM – 5:30AM
Affected services: Websites or Apps relying on databases on mysql01
Affected systems: mysql01
Description of maintenance: We are making changes to simplify the data backup and recovery process for the mysql database server.
User impact: Websites or apps relying on databases located on mysql01 will be unavailable during the maintenance window.
When: 8:00AM – 2:00PM
Affected services: Echo360
Affected systems: Echo360
Description of maintenance: The Echo class video system is being updated to improve response client times with the system. Along with the software update to 5.3 SP2, we will also be separating the web services component of the Echo360 system to a new virtualized environment. Currently the Echo system management server is also hosting the web services for this system. Separating out this service will allow the system to better handle client viewing traffic load.
User impact: End-user impact will be minimized at this time, as regular term classes will no longer access content stored in the Echo system. However the College of Medicine has ongoing classes that do use the Echo360 system to distribute class video content; they confirmed this date/time is best for their class schedule.
Echo Player streaming class videos as well as audio/video podcast downloads will be available intermittently during this maintenance window.
Also, the Echo management interface at https://echo.fau.edu will be intermittently available during this time. This will affect the publishing of Echo Personal Capture and External Media Ingest videos. Because of this, we recommend postponing the publishing of any Personal Capture or External Media.
Office of Information Technology will be performing routine systems maintenance which includes but is not limited to rebooting Windows servers as deemed necessary, installing critical patches and other health checks on Wednesday, 11/27/2013 from 3:00 AM to 7:00 AM during OIT maintenance window. In addition, the following systems will be specifically impacted:
When: 3am to 6am.
Affected Services: IDM production, Luminis production, Phonebill app, Report Caster production, IEA Production database IEAP, Legacy Travel prod database FINP.
Affected Systems: boc22ora01, boc22ora03,boc22ora05 , mcoracle, nwrdcora01.
User Impact: Affected services will be unavailable during the maintenance window.
Reason: Oracle database patch set updates and host patch installations
Communications Infrastructure will be performing network maintenance from 3:00a.m. until 3:30a.m. on Wednesday November 26th to resolve issues with network slowness. During this time there may be a short network outage lasting approximately 5 minutes. If such an outage occurs, this outage will affect connectivity between FAU and the internet. This maintenance is expected to temporarily resolve the following issues that have been reported by end users:
Frequent pauses in Echo360 recordings
Poor Voice-over-IP quality (Non-FAU VoIP providers such as Skype)
Slow / Laggy performance of Remote Desktop sessions for VPN users
Slow / Laggy performance in online games from on-campus housing
RESOLVED: Echo360 will be intermittently unavailable from 3:30-4:00 pm. The Echo360 service needs to be restarted to free a blocked thread that is preventing content from being ingested. Any user viewing an Echo recording will have their service interrupted while the item restarts.
The service has been restarted and captures are now being routed to the processing queue as designed.
OIT has identified a storage problem which was the cause for the service outage. The problem has been corrected and services are returning to their normal state. We thank you for your patience.
OIT has experienced a problem in its data center which are affecting multiple services, some of which are:
– DNS
– Blackboard
– Sharepoint
– Internal websites
We are investigating the problem and will post another update at 7:30am. We appreciate your patience.